[issue35813] shared memory construct to avoid need for serialization between processes
Philip Semanchuk added the comment:
> On Feb 23, 2019, at 10:40 AM, Giampaolo Rodola'
> wrote:
>
>
> Giampaolo Rodola' added the comment:
>
>> We are consciously choosing to not support an atomic "create or attach".
>> This significantly simplifies the API and avoids the valid concerns raised
>> around user confusion relating to that behavior (including the use of
>> different specified 'size' values in a race) but does not preclude our
>> potentially introducing this as a feature in the future.
>
> I understand that because of *size* we cannot solve the race condition issue
> unless the user uses some sort of synchronization mechanism. FWIW I bumped
> into this lib:
> http://semanchuk.com/philip/sysv_ipc/
> ...which provides two separate APIs to "create" and "attach":
>
>>>> SharedMemory("name", IPC_CREX)
>>>> attach("name")
>
> At this point I'm agnostic about the API, which is probably just a matter of
> personal taste (e.g. one may prefer a separate SharedMemory.attach()
> classmethod or a *mode* argument accepting "x" and "a"). I see that that lib
> use shmat() on attach and shmdt() on detach. I'm not sure if that makes a
> difference, just mentioning it because your implementation doesn't do that on
> close() and perhaps it should.
attach() and detach() are particular to SysV IPC which is different from the
POSIX IPC that’s being used here. There’s no need for attach() and detach()
with POSIX shared memory.
POSIX IPC is generally simpler than SysV IPC, in part because it was developed
after SysV IPC so the developers had the benefit of experience with the older
API.
Side note: I’m the author of the sysv_ipc package you found, as well as the
posix_ipc package.
--
___
Python tracker
<https://bugs.python.org/issue35813>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Philip Semanchuk added the comment: Hi all, I'm the author of `posix_ipc` on which some of this code is based. I'd be happy to sign a contributor agreement in order to erase any concerns on that front. -- nosy: +osvenskan ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20584] On FreeBSD, signal.NSIG is smaller than biggest signal value
Changes by Philip Semanchuk osvens...@users.sourceforge.net: -- nosy: +osvenskan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20584 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Need help with file encoding-decoding
On Sep 23, 2011, at 7:44 AM, Yaşar Arabacı wrote: Hi, I'am trying to write a mass html downloader, and it processes files after it downloaded them. I have problems with encodings, and decodings. Sometimes I get UnicodeDecodeErrors, or I get half-pages in after processing part. Or more generally, some things don't feel right. Can you check my approach, and provide me some feedback please? Here is what I am doing. 1) send a HEAD request to file's source to get file encoding, set encoding variable accordingly. Hi Yaşar This is a pretty optimistic algorithm, at least by the statistics from 2008 (see below). 2) if server doesn't provide an encoding, set encoding variable as utf-8 This is statistically a good guess but it doesn't follow the HTTP specification. 4) in this step, I need to parse the content I get, because I will search for further links \ I feed content to parser (subclass of HTMLParser.HTMLParser) like Does HTMLParser.HTMLParser handle broken HTML? Because there's lots of it out there. I used to run an automated site validator, and I wrote a couple of articles you might find interesting. One is about how to get the encoding of a Web page: http://NikitaTheSpider.com/articles/EncodingDivination.html I also wrote an article examining the statistics I'd seen run through the crawler/validator. One thing I saw was that almost 2/3 of Web pages specified the encoding in the META HTTP-EQUIV Content-Type tag rather than in the HTTP Content-Type header. Mind you, this was three years ago so the character of the Web has likely changed since then, but probably not too dramatically. http://NikitaTheSpider.com/articles/ByTheNumbers/fall2008.html You can also do some straightforward debugging. Save the raw bytes you get from each site, and when you encounter a decode error, check the raw bytes. Are they really in the encoding specified? Webmasters make all kinds of mistakes. Hope this helps Philip this - content.decode(encoding) 5) open a file in binary mod open(file_path,wb) 6) I write as I read without modifing. ## # After processing part ## (Note: encoding variable is same as the downloading part) 1) open local file in binary mod for reading file_name = open(file_path,rb) 2) decode the file contents into a variable = decoded_content = file_name.read().decode(encoding) 3) send decoded content to a parser, parser contstruct new html content. (as str) 4) open same file for writing, in binary mod, write parsers output like this: file_name.write(parser.output.encode(encoding)) -- http://yasar.serveblog.net/ -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Numpy.array with dtype works on list of tuples not on list of lists?
On Sep 18, 2011, at 11:55 AM, Alex van der Spek wrote:
Why does this not work?
dat=[[1,2,3],[4,5,6]]
col=[('a','f4'),('b','f4'),('c','f4')]
arr=numpy.array(dat,dtype=col)
Traceback (most recent call last):
File pyshell#91, line 1, in module
arr=numpy.array(dat,dtype=col)
TypeError: expected a readable buffer object
But this does:
dat=[(1,2,3),(4,5,6)]
arr=numpy.array(dat,dtype=col)
arr
array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0)], dtype=[('a', 'f4'), ('b',
'f4'), ('c', 'f4')])
The only difference that the object is a list of tuples now?
I don't know why you're seeing what you're seeing, but if you don't get answer
here you could try asking on the numpy list.
Good luck
Philip
--
http://mail.python.org/mailman/listinfo/python-list
Python Tools for Visual Studio - anyone using it?
Hi all, I was reminded today (via Slashdot) of Python Tools for Visual Studio which was discussed on this list back in March (http://mail.python.org/pipermail/python-list/2011-March/1267662.html) and has reached version 1.0. Is anyone here using it? Care to share pros cons? Here's the URL for those who haven't heard of it before: http://pytools.codeplex.com/ Thanks Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Help parsing a text file
On Aug 29, 2011, at 2:21 PM, William Gill wrote: I haven't done much with Python for a couple years, bouncing around between other languages and scripts as needs suggest, so I have some minor difficulty keeping Python functionality Python functionality in my head, but I can overcome that as the cobwebs clear. Though I do seem to keep tripping over the same Py2 - Py3 syntax changes (old habits die hard). I have a text file with XML like records that I need to parse. By XML like I mean records have proper opening and closing tags. but fields don't have closing tags (they rely on line ends). Not all fields appear in all records, but they do adhere to a defined sequence. My initial passes into Python have been very unfocused (a scatter gun of too many possible directions, yielding very messy results), so I'm asking for some suggestions, or algorithms (possibly even examples)that may help me focus. I'm not asking anyone to write my code, just to nudge me toward a more disciplined approach to a common task, and I promise to put in the effort to understand the underlying fundamentals. If the syntax really is close to XML, would it be all that difficult to convert it to proper XML? Then you have nice libraries like ElementTree to use for parsing. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Button Label change on EVT_BUTTON in wxpython!!!
On Aug 28, 2011, at 9:30 PM, Ven wrote:
Some system info before proceeding further:
Platform: Mac OS X 10.7.1
Python Version: ActiveState Python 2.7.1
wxPython Version: [url=http://downloads.sourceforge.net/wxpython/
wxPython2.9-osx-2.9.2.1-cocoa-py2.7.dmg]wxPython2.9-osx-cocoa-py2.7[/
url]
I want the button label to be changed while performing a task
So, here is what I did/want:
self.run_button=wx.Button(self.panel,ID_RUN_BUTTON,label='Install')
self.Bind(wx.EVT_BUTTON, self.OnRun,id=ID_RUN_BUTTON)
def OnRun(self,evt):
self.run_button.SetLabel('Installing..')
#call a function that does the installation task
installation_task()
#After task completion, set the button label back to Install
self.run_button.SetLabel('Install')
When I try doing this, it doesn't set the label to Installing while
the task is being performed. Any suggestions how do I achieve this?
Suggestion #1: After you set the label to Installing..., try adding
self.run_button.Refresh() and/or self.run_button.Update().
Suggestion #2: Ask wxPython questions on the wxPython mailing list.
Good luck
Philip
--
http://mail.python.org/mailman/listinfo/python-list
Re: Understanding .pth in site-packages
On Aug 27, 2011, at 12:56 PM, Josh English wrote: (This may be a shortened double post) I have a development version of a library in c:\dev\XmlDB\xmldb After testing the setup script I also have c:\python27\lib\site-packages\xmldb Now I'm continuing to develop it and simultaneously building an application with it. I thought I could plug into my site-packages directory a file called xmldb.pth with: c:\dev\XmlDB\xmldb which should redirect import statements to the development version of the library. This doesn't seem to work. xmldb.pth should contain the directory that contains xmldb: c:\dev\XmlDB Examining sys.path at runtime probably would have helped you to debug the effect of your .pth file. On another note, I don't know if the behavior of 'import xmldb' is defined when xmldb is present both as a directory in site-pacakges and also as a .pth file. You're essentially giving Python two choices from where to import xmldb, and I don't know which Python will choose. It may be arbitrary. I've looked for some sort of statement on this topic in the documentation, but haven't come across it yet. Is there a better way to redirect import statements without messing with the system path or the PYTHONPATH variable? Personally I have never used PYTHONPATH. Hope this helps Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Understanding .pth in site-packages
On Aug 27, 2011, at 1:57 PM, Josh English wrote: Philip, Yes, the proper path should be c:\dev\XmlDB, which has the setup.py, xmldb subfolder, the docs subfolder, and example subfolder, and the other text files proscribed by the package development folder. I could only get it to work, though, by renaming the xmldb folder in the site-packages directory, and deleting the egg file created in the site-packages directory. Why the egg file, which doesn't list any paths, would interfere I do not know. But with those changes, the xmldb.pth file is being read. So I think the preferred search order is: 1. a folder in the site-packages directory 2. an Egg file (still unsure why) 3. A .pth file That might be implementation-dependent or it might even come down to something as simple as the in which order the operating system returns files/directories when asked for a listing. In other words, unless you can find something in the documentation (or Python's import implementation) that confirms your preferred search order observation, I would not count on it working the same way with all systems, all Pythons, or even all directory names. Good luck Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Understanding .pth in site-packages
On Aug 27, 2011, at 4:14 PM, Terry Reedy wrote: On 8/27/2011 2:07 PM, Philip Semanchuk wrote: On Aug 27, 2011, at 1:57 PM, Josh English wrote: Philip, Yes, the proper path should be c:\dev\XmlDB, which has the setup.py, xmldb subfolder, the docs subfolder, and example subfolder, and the other text files proscribed by the package development folder. I could only get it to work, though, by renaming the xmldb folder in the site-packages directory, and deleting the egg file created in the site-packages directory. Why the egg file, which doesn't list any paths, would interfere I do not know. But with those changes, the xmldb.pth file is being read. So I think the preferred search order is: 1. a folder in the site-packages directory 2. an Egg file (still unsure why) 3. A .pth file That might be implementation-dependent or it might even come down to something as simple as the in which order the operating system returns files/directories when asked for a listing. Doc says first match, and I presume that includes first match within a directory. First match using which ordering? Do the docs clarify that? Thanks Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Understanding .pth in site-packages
On Aug 27, 2011, at 6:49 PM, Josh English wrote:
When I run: os.listdir('c:\Python27\lib\site-packages') I get the contents in
order, so the folders come before .pth files (as nothing comes before
something.)
That's one definition of in order. =)
I would guess Python is using os.listdir. Why wouldn't it?
If you mean that Python uses os.listdir() during import resolution, then yes I
agree that's probable. And os.listdir() doesn't guarantee any consistent order.
In fact, the documentation explicitly states that the list is returned in
arbitrary order. Like a lot of things in Python, os.listdir() probably relies
on the underlying C library which varies from system to system. (Case in point
-- on my Mac, os.listdir() returns things in the same order as the 'ls'
command, which is case-sensitive alphabetical, files directories mixed --
different from Windows.)
So if import relies on os.listdir(), then you're relying on arbitrary
resolution when you have a .pth file that shadows a site-packages directory.
Those rules will probably work consistently on your particular system, you're
developing a habit around what is essentially an implementation quirk.
Cheers
Philip
--
http://mail.python.org/mailman/listinfo/python-list
Re: Immediate Requirement for a Data Warehouse Developer
On Aug 25, 2011, at 9:24 AM, Sirisha wrote: Position Profile – Senior Data Warehouse Developer As was mentioned on the list less than 24 hours ago, please don't post job listings to this mailing list. Use the Python jobs board instead: http://www.python.org/community/jobs/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Hot Girls are Looking for Sex
On Aug 19, 2011, at 4:17 PM, Matty Sarro wrote: That's great - but do they program in python? Please don't repost URLs sent by a spammer. Only Google truly knows how its algorithm works, but the general consensus is that the more times Google sees a link repeated, the more credibility the link is given. By reposting links, you help the spammer. -- http://mail.python.org/mailman/listinfo/python-list
Re: List spam
On Aug 18, 2011, at 8:58 AM, Jason Staudenmayer wrote: I really like this list as part of my learning tools but the amount of spam that I've been getting from it is CRAZY. Doesn't anything get scanned before it sent to the list? This has been discussed on the list a number of times before, so I'll refer you to the archives for details. Basically, the mailing list receives postings from Google Groups and vice versa. Most of the spam comes from Google Groups. If you add a mail filter that deletes anything with the Organization header set to http://groups.google.com;, you won't see much spam anymore. In my experience, you'll also miss a number of legitimate postings. HTH Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: List spam
On Aug 18, 2011, at 1:10 PM, Peter Pearson wrote: On Thu, 18 Aug 2011 12:15:59 -0400, gene heskett ghesk...@wdtv.com wrote: [snip] What is wrong with the mailing list only approach? In the mailing-list approach, how do I search for prior discussions on a subject? (I'm not particularly opposed to the mailing list, I'm just an NNTP follower worried about the uncertainties of change.) I use a Google search like this: site:mail.python.org/pipermail/python-list/ banana Although that has its own issues, as not all messages seem to make it to that list (or they have the X-No-Archive bit set?) Cheers P -- http://mail.python.org/mailman/listinfo/python-list
Re: Why no warnings when re-assigning builtin names?
On Aug 16, 2011, at 1:15 AM, Steven D'Aprano wrote: On Tue, 16 Aug 2011 01:23 pm Philip Semanchuk wrote: On Aug 15, 2011, at 9:32 PM, Steven D'Aprano wrote: On Tue, 16 Aug 2011 08:15 am Chris Angelico wrote: If you want a future directive that deals with it, I'd do it the other way - from __future__ import mask_builtin_warning or something - so the default remains as it currently is. But this may be a better job for a linting script. Agreed. It's a style issue, nothing else. There's nothing worse about: def spam(list): pass compared to class thingy: pass def spam(thingy): pass Why should built-ins be treated as more sacred than your own objects? Because built-ins are described in the official documentation as having a specific behavior, while my objects are not. *My* objects certainly are, because I write documentation for my code. My docs are no less official than Python's docs. I'm sure they are no less official to you. But you are you, and then there's...everyone else. =) I (and I think most people) give far more credibility to the Python docs than to the documentation of an individual. That's not a reflection on you, it reflects the limits of one person's ability versus organizationally produced docs which are heavily used, discussed, and have been iteratively developed over many years. Sometimes shadowing is safe, sometimes it isn't. Sometimes X is safe and sometimes it isn't can be said of many, many things, from taking a walk down the street to juggling with knives. But it has little to do with whether or not Python should issue a warning in the specific case we're talking about. A warning that is off by default won't help the people who need it, because they don't know enough to turn the warning on. I agree that it wouldn't help the people who need it most (absolute raw newcomers). But you're asserting that once one learned the incantation to enable the theoretical warning we're discussing, one would have graduated to a level where it's no longer useful. That's not the case. There's a lot of ground to cover between newcomer who has learned about a particular warning and coder who regularly shadows builtins on purpose. I am an example. I know enough to turn the theoretical warning on, and I would if I could. I have never shadowed a builtin deliberately. I've done it accidentally plenty of times. There are 84 builtins in my version of Python and I don't have them all memorized. The fact that my editor colors them differently is the only thing I have to back up my leaky memory. Not all editors are so gracious. Yes, it can be useful to replace some of the builtins with one's own implementation, and yes, doing so fits in with Python's we're all consenting adults philosophy. But replacing (shadowing, masking -- call it what you will) builtins is not everyday practice. On the contrary, as the OP Gerrat pointed out, it's most often done unwittingly by newcomers to the language who have no idea that they've done anything out of the ordinary or potentially confusing. Protecting n00bs from their own errors is an admirable aim, but have you considered that warnings for something which may be harmless could do more harm than good? Isn't the whole point of a warning to highlight behavior that's not strictly wrong but looks iffy? Sort of, I can't be sure, but this looks like trouble to me. I hope you know what you're doing. If we are to eschew warnings in cases where they might be highlighting something harmless, then we would have no warnings at all. Again, shadowing builtins is not everyday practice. I have been trying to remember if I've ever seen it done deliberately, and I can't remember a case. Now, a comment like that is an invitation for people come out of the woodwork with cases where they found it useful, and I would welcome some examples as I'm sure they'd be interesting. But I think it's safe to say that if you look at random samples of code, builtins are shadowed unintentionally hundreds of times for every time they're shadowed deliberately and usefully. If a language feature is most often invoked accidentally without knowledge of or regard for its potential negative consequences, then it might be worth making it easier to avoid those accidents. Perhaps. But I'm not so sure it is worth the cost of extra code to detect shadowing and raise a warning. After all, the average coder probably never shadows anything, One need look no further than the standard library to see a strong counterexample. grep through the Python source for file =. I see dozens of examples of this builtin being used as a common variable name. I would call contributors to the standard library above-average coders, and we can see them unintentionally shadowing builtins many times. and for those that do, once they get bitten *once* they either never do it again or learn how to shadow safely. I have done
Re: Why no warnings when re-assigning builtin names?
On Aug 16, 2011, at 11:12 AM, Chris Angelico wrote: On Tue, Aug 16, 2011 at 3:13 PM, Philip Semanchuk phi...@semanchuk.com wrote: One need look no further than the standard library to see a strong counterexample. grep through the Python source for file =. I see dozens of examples of this builtin being used as a common variable name. I would call contributors to the standard library above-average coders, and we can see them unintentionally shadowing builtins many times. There are several types of shadowing: 1) Deliberate shadowing because you want to change the behavior of the name. Extremely rare. 2) Shadowing simply by using the name of an unusual builtin (like 'file') in a context where you never use it. Very common. 3) Unintentional shadowing where you create a variable, but then intend to use the builtin. This is the only one that's a problem. Yes, but before you get to #3 you have to go through #2. The way I see it, #2 is setting a trap, #3 is actually stepping in it. I don't want to do either. Neither do I like working with code that has set trap #2 for me. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Why no warnings when re-assigning builtin names?
On Aug 16, 2011, at 11:41 AM, Ethan Furman wrote: Philip Semanchuk wrote: On Aug 16, 2011, at 1:15 AM, Steven D'Aprano wrote: Protecting n00bs from their own errors is an admirable aim, but have you considered that warnings for something which may be harmless could do more harm than good? Isn't the whole point of a warning to highlight behavior that's not strictly wrong but looks iffy? Sort of, I can't be sure, but this looks like trouble to me. I hope you know what you're doing. If we are to eschew warnings in cases where they might be highlighting something harmless, then we would have no warnings at all. Sounds good to me. ;) Keep such things in the IDE's, and then those who desire such behavior can have it there. Do not clutter Python with such. You wink, yet you sound serious. What's with the mixed message? Do you honestly advocate removing all warnings from Python, or not? I sincerely would like to know what you think. Perhaps. But I'm not so sure it is worth the cost of extra code to detect shadowing and raise a warning. After all, the average coder probably never shadows anything, One need look no further than the standard library to see a strong counterexample. grep through the Python source for file =. I see dozens of examples of this builtin being used as a common variable name. I would call contributors to the standard library above-average coders, and we can see them unintentionally shadowing builtins many times. What makes you think it's unintentional? file makes a good variable name, and if you don't need it to actually open a file there's nothing wrong with using it yourself. Unintentional as in, I'm using file as a variable name because it's handy as opposed to intentional as in Yes, I am deliberately changing the meaning of this builtin. and for those that do, once they get bitten *once* they either never do it again or learn how to shadow safely. I have done it plenty of times, never been bitten (thankfully) and still do it by accident now and again. Seems to me the real issue is somebody using a builtin, such as str or int, and that they somehow manage to do this without realizing, wait a sec', that's one of my variables! Yes I don't see that as a problem that Python needs to solve. need is a strong word. Python will be fine regardless of whether this changes or not. I believe Python could be improved; that's all I'm arguing. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Why no warnings when re-assigning builtin names?
On Aug 16, 2011, at 12:19 PM, Ethan Furman wrote: Philip Semanchuk wrote: On Aug 16, 2011, at 11:41 AM, Ethan Furman wrote: Philip Semanchuk wrote: If we are to eschew warnings in cases where they might be highlighting something harmless, then we would have no warnings at all. Sounds good to me. ;) Keep such things in the IDE's, and then those who desire such behavior can have it there. Do not clutter Python with such. You wink, yet you sound serious. The smiley is an attempt to not sound harsh. Thanks. It's hard to know on the Internet. I don't see that as a problem that Python needs to solve. need is a strong word. Python will be fine regardless of whether this changes or not. I believe Python could be improved; that's all I'm arguing. Python can be improved -- I don't see 'hand-holding' as an improvement. IDEs and lints can do this. When you say hand-holding, I hear a pejorative. That makes I don't see 'hand-holding' as an improvement a tautology. Have I misheard you? I think Python does lots of beneficial hand-holding. Garbage collection is a good example. $DIETY knows, people have been struggling with manual memory management in C and its ilk for a long time. Even though there are good tools to help, memory leaks still happen. Python increases our productivity by allowing us to forget about manual memory management altogether. I can do it with tools like valgrind, but Python's makes the point moot. Is that hand-holding? If so, I'm all for it. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Why no warnings when re-assigning builtin names?
On Aug 16, 2011, at 7:29 PM, Terry Reedy wrote: On 8/16/2011 1:15 PM, Gerrat Rickert wrote: I think that best practices would suggest that one shouldn't use variable names that shadow builtins (except in specific, special circumstances), so I don't really think this would be an annoyance at all. The number of *unwanted* warnings they'd get would be pretty close to zero. OTOH, in response to a question I asked on StackOverflow, someone posted a large list of times where this isn't followed in the std lib, so there seems to be a precedent for just using the builtin names for anything one feels like at the time. If you run across that again and email me the link, I will take a look and see if I think the issue should be raised on pydev. Of course, some modules *intentionally* define an open function, intended to be accessed as 'mod.open' and not as 'from mod import *; open'. Also, class/instance attributes can also reuse builtin names. But 'open = True/False' would be bad. Hi Terry, To generalize from your example, are you saying that there's a mild admonition against shadowing builtins with unrelated variable names in standard lib code? Here's an example from Python 3.2.1's argparse.py, lines 466-473. open is shadowed on the second line. # clean up separators for mutually exclusive groups open = r'[\[(]' close = r'[\])]' text = _re.sub(r'(%s) ' % open, r'\1', text) text = _re.sub(r' (%s)' % close, r'\1', text) text = _re.sub(r'%s *%s' % (open, close), r'', text) text = _re.sub(r'\(([^|]*)\)', r'\1', text) text = text.strip() Thanks Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Why no warnings when re-assigning builtin names?
On Aug 16, 2011, at 9:29 PM, Steven D'Aprano wrote: I have no objection to lint tools. But separation of concerns should apply: the Python compiler should just compile what I tell it to, the linter should warn me if I'm running with scissors. This point (also made by Ethan) I can agree with. I haven't looked through all the warnings the Python compiler emits, but it seems like it currently doesn't dispense advice (unlike, say, gcc). It only warns about changes in the language standard library. In that context, asking it to warn about shadowing builtins would be an expansion of scope. bye, Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Why no warnings when re-assigning builtin names?
On Aug 16, 2011, at 10:15 PM, Terry Reedy wrote: On 8/16/2011 8:18 PM, Philip Semanchuk wrote: Hi Terry, To generalize from your example, are you saying that there's a mild admonition against shadowing builtins with unrelated variable names in standard lib code? I would expect that there might be. I would have to check PEP8. I was curious, so I checked. I didn't see anything specifically referring to builtins. This is as close as it gets: If a function argument's name clashes with a reserved keyword, it is generally better to append a single trailing underscore rather than use an abbreviation or spelling corruption. Thus print_ is better than prnt. (Perhaps better is to avoid such clashes by using a synonym.) bye Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Help needed with using SWIG wrapped code in Python
On Aug 15, 2011, at 4:08 AM, Vipul Raheja wrote: Hi, I have wrapped a library from C++ to Python using SWIG. But I am facing problems while importing and using it in Python. Hi Vipul, Did you try asking about this on the SWIG mailing list? bye Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Why no warnings when re-assigning builtin names?
On Aug 15, 2011, at 5:52 PM, Gerrat Rickert wrote: With surprising regularity, I see program postings (eg. on StackOverflow) from inexperienced Python users accidentally re-assigning built-in names. For example, they'll innocently call some variable, list, and assign a list of items to it. ...and if they're _unlucky_ enough, their program may actually work (encouraging them to re-use this name in other programs). Or they'll assign a class instance to 'object', only to cause weird errors later when they use it as a base class. I agree that this is a problem. The folks on my project who are new-ish to Python overwrite builtins fairly often. Since there's never been any consequence other than my my vague warnings that something bad might happen as a result, it's difficult for them to develop good habits in this regard. It doesn't help that Eclipse (their editor of choice) doesn't seem to provide a way of coloring builtins differently. (That's what I'm told, anyway. I don't use it.) If they try to use an actual keyword, both the interpreter and compiler are helpful enough to give them a syntax error, but I think the builtins should be pseudo-reserved, and a user should explicitly have to do something *extra* to not receive a warning. Unfortunately you're suggesting a change to the language which could break existing code. I could see a use for from __future__ import squawk_if_i_reassign_a_builtin or something like that, but the current default behavior has to remain as it is. JMO, Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Why no warnings when re-assigning builtin names?
On Aug 15, 2011, at 9:32 PM, Steven D'Aprano wrote: On Tue, 16 Aug 2011 08:15 am Chris Angelico wrote: If you want a future directive that deals with it, I'd do it the other way - from __future__ import mask_builtin_warning or something - so the default remains as it currently is. But this may be a better job for a linting script. Agreed. It's a style issue, nothing else. There's nothing worse about: def spam(list): pass compared to class thingy: pass def spam(thingy): pass Why should built-ins be treated as more sacred than your own objects? Because built-ins are described in the official documentation as having a specific behavior, while my objects are not. Yes, it can be useful to replace some of the builtins with one's own implementation, and yes, doing so fits in with Python's we're all consenting adults philosophy. But replacing (shadowing, masking -- call it what you will) builtins is not everyday practice. On the contrary, as the OP Gerrat pointed out, it's most often done unwittingly by newcomers to the language who have no idea that they've done anything out of the ordinary or potentially confusing. If a language feature is most often invoked accidentally without knowledge of or regard for its potential negative consequences, then it might be worth making it easier to avoid those accidents. bye, Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing timing issue
On Aug 9, 2011, at 1:07 PM, Tim Arnold wrote: Hi, I'm having problems with an empty Queue using multiprocessing. The task: I have a bunch of chapters that I want to gather data on individually and then update a report database with the results. I'm using multiprocessing to do the data-gathering simultaneously. Each chapter report gets put on a Queue in their separate processes. Then each report gets picked off the queue and the report database is updated with the results. My problem is that sometimes the Queue is empty and I guess it's because the get_data() method takes a lot of time. I've used multiprocessing before, but never with a Queue like this. Any notes or suggestions are very welcome. Hi Tim, THis might be a dumb question, but...why is it a problem if the queue is empty? It sounds like you figured out already that get_data() sometimes takes longer than your timeout. So either increase your timeout or learn to live with the fact that the queue is sometimes empty. I don't mean to be rude, I just don't understand the problem. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: WxPython and TK
On Aug 7, 2011, at 8:26 PM, azrael wrote: Today I found a quote from Guido. wxPython is the best and most mature cross-platform GUI toolkit, given a number of constraints. The only reason wxPython isn't the standard Python GUI toolkit is that Tkinter was there first. -- Guido van Rossum OK, now. Isn't it maybe time to throw out TK once and for all? Python is missing one of the most important aspects of todays IT industry. GUI development native library (I mean a serious one). I don't see how removing TK from the standard library helps to fill the native GUI development library void that you see in Python. I guess you're promoting wxPython as the library to fill that void. Getting rid of TK is one argument, adding wxPython is different argument. Are you advocating one, the other, or both? If I would have gotten a dollar for every time I talked to someone in a company about why they dont use python for their products and I was served the answer Well it kind of sucks in GUI development, I would be a millionaire. And if I had a dollar for every Let's replace TK with XYZ post, I'd also be a millionaire. I don't object to your argument; criticism of standard library is how it advances. But you're going to have to come up with a better argument than a 5+ year old quote from Guido and an exaggerated claim about why people don't use Python. The best Python GUI library conversation is repeated on this list at least once every few months. If the subject really interests you, I recommend that you read the archives and see some of the arguments for and against various GUI toolkits. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Table Driven GUI Definition?
On Aug 5, 2011, at 4:10 PM, Tim Daneliuk wrote: On 8/5/2011 2:05 PM, Irmen de Jong said this: On 05-08-11 19:53, Tim Daneliuk wrote: I have a task where I want to create pretty simple one page visual interfaces (Graphical or Text, but it needs to run across Windows, Cygwin, Linux,*BSD, OSX ...). These interfaces are nothing more than option checklists and text fields. Conceptually something like: Please Select Your Installation Options: Windows Compatibility Services _ Linux Compatibility Services_ TRS-DOS Compatibility Services _ What Is Your email Address: ___ What I'm looking for is a way to describe such forms in a text file that can then be fed into a tool to generate the necessary pyGUI, Tkinter, (or whatever) code. The idea is that it should be simple to generate a basic interface like this and have it only record the user's input. Thereafter, the python code would act on the basis of those selection without any further connection to the GUI. An added bonus would be a similar kind of thing for generating web interfaces to do this. This might actually be a better model because then I only have to worry about a single presentation environment. Ideas anyone? Hi Tim This looks pretty straightforward to me; maybe I'm missing something. It doesn't look trivial, but the steps seem pretty clear. Is there some part in particular that's giving you trouble? Cheers Philip Yeah, HTML being the text file and a web browser being the tool to transform it into a GUI... You can hook this up with a simple web server or web framework running locally to grab the submitted form results when the form is complete and process them in a piece of python code. Wouldn't that work? Irmen Yup, although I'd probably use a central apache instance. But I'm still curious ... is there a way to do this with a full GUI tool on a thick client? -- Tim Daneliuk tun...@tundraware.com -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Table Driven GUI Definition?
On Aug 5, 2011, at 6:20 PM, Tim Daneliuk wrote: On 8/5/2011 3:42 PM, Philip Semanchuk wrote: On Aug 5, 2011, at 4:10 PM, Tim Daneliuk wrote: On 8/5/2011 2:05 PM, Irmen de Jong said this: On 05-08-11 19:53, Tim Daneliuk wrote: I have a task where I want to create pretty simple one page visual interfaces (Graphical or Text, but it needs to run across Windows, Cygwin, Linux,*BSD, OSX ...). These interfaces are nothing more than option checklists and text fields. Conceptually something like: Please Select Your Installation Options: Windows Compatibility Services _ Linux Compatibility Services_ TRS-DOS Compatibility Services _ What Is Your email Address: ___ What I'm looking for is a way to describe such forms in a text file that can then be fed into a tool to generate the necessary pyGUI, Tkinter, (or whatever) code. The idea is that it should be simple to generate a basic interface like this and have it only record the user's input. Thereafter, the python code would act on the basis of those selection without any further connection to the GUI. An added bonus would be a similar kind of thing for generating web interfaces to do this. This might actually be a better model because then I only have to worry about a single presentation environment. Ideas anyone? Hi Tim This looks pretty straightforward to me; maybe I'm missing something. It doesn't look trivial, but the steps seem pretty clear. Is there some part in particular that's giving you trouble? Cheers Philip I want to take a text definition file that looks something this: Title Please Select Your Installation Options: Checkbox Windows Compatibility Services Checkbox Linux Compatibility Services Checkbox TRS-DOS Compatibility Services Inputbox What Is Your email Address: And have that aut-generate the GUI interface described above for the selected GUI toolkit and/or an equivalent HTML page. I know I can write a program to do this, but it seems that someone else may have already solved this problem. Oh, I see. I didn't realize you were looking for a most canned solution. I agree that it's a problem that's been solved many times. I've used Mako before as an HTML templating engine, but ISTR that it points out that it's agnostic to what it's templating. In other words, it only cares about what's between the Mako escape tags, it doesn't care if the surrounding text is HTML or XML or Python or whatever. So you could have a Mako template that consists mostly of Python code that builds a wxPython window (if wxPython is your cup of tea) and then some Mako commands in the middle that reads your text definition file and adds checkboxes, textboxes, etc. as appropriate. It's not a canned solution, but it does allow you to separate the boilerplate stuff from the variants. Hope this helps Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Deeply nested dictionaries - should I look into a database or am I just doing it wrong?
On Jul 31, 2011, at 4:04 PM, Thorsten Kampe wrote: * Andrew Berg (Sun, 31 Jul 2011 13:36:43 -0500) On 2011.07.31 02:41 AM, Thorsten Kampe wrote: Another approach would be named tuples instead of dictionaries or flat SQL tables. What would the advantage of that be? QueueItem.x264['avs']['filter']['fft3d']['ffte'] would be QueueItem.x264.avs.filter.fft3d.ffte. I recently migrated from a syntax of - example - datetuple[fieldpositions['tm_year'][0]] (where fieldpositions was a dictionary containing a list) to datetuple.tm_year_start which is much more readable. The advantage of a SQL(ite) database would be simple flat tables but accessing them would be more difficult. Even a INI config file structure could match your problem. INI files are OK for lightweight use, but I find them very fragile. Since there's no specification for them, libraries don't always agree on how to read them. For instance, some libraries treat # as the comment character, and others think it is ; and others accept both. There's no standard way to specify the encoding, and, as would be critical to the OP who is nesting dicts inside of dicts, not all INI file libraries accept nested sections. To the OP -- if you're looking to write this to disk, I recommend XML or SQLite. JMHO, Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: how to get Python to insert special characters in an xml file?
On Jul 15, 2011, at 7:53 AM, hackingKK wrote: Hello all. I am currently developing a python application which reads and writes some data to an xml file. I use the elementTree library for doing this. My simple question is that if I have some thing like as in kk company as organisation name, how can I have Python take this as a litteral string including the sign and put in the orgname /orgname tag? Even same applies while reading the file. I would like to have the come as a part of the literal string. Hi Krishnakant, You don't need to do anything special to insert metacharacters like and and into XML using ElementTree. Just treat them as normal text and ElementTree will change them to entity references (amp;, etc.) when it writes your file to disk. If you're having a specific problem with this, post some code. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: wx MenuItem - icon is missing
On Jul 6, 2011, at 2:25 AM, Laszlo Nagy wrote: Under windows, this displays the icon for the popup menu item. Under GTK it doesn't and there is no error message, no exception. I get different results than you. Under Ubuntu 9.04 w with wx 2.8.9.1, when I right click I see a menu item called test with little icon of a calculator or something. Under OS X 10.6 with wx 2.8.12.0 and Win XP with wx 2.8.10.1, when I right click I get this -- Traceback (most recent call last): File x.py, line 46, in onPopupMenu item = wx.MenuItem(None,-1,uTest) File /usr/local/lib/wxPython-unicode-2.8.12.0/lib/python2.6/site-packages/wx-2.8-mac-unicode/wx/_core.py, line 11481, in __init__ _core_.MenuItem_swiginit(self,_core_.new_MenuItem(*args, **kwargs)) wx._core.PyAssertionError: C++ assertion parentMenu != NULL failed at /BUILD/wxPython-src-2.8.12.0/src/common/menucmn.cpp(389) in wxMenuItemBase(): menuitem should have a menu I guess I'll have to write to the wxPython mailing list. Seriously, adding a simple menu to something is supposed to be platform independent, but we got four different results on four systems. :-( I can understand why it's frustrating but a menu items with icons on them aren't exactly common, so you're wandering into territory that's probably not so throughly explored (nor standard across platforms). Now that I think about it, I don't know that I've ever seen one under OSX, and I don't even know if it's supported at all. Me, I would start by addressing the error in the traceback. wx doesn't seem happy with an orphan menu item; why not create a wx.Menu and assign the menu item to that? It might solve your icon problem; you never know. In defense of wxPython, we have three wx apps in our project and they contain very little platform-specific code. To be fair, we've had to rewrite some code after we found that it worked on one platform but not another, but generally we're able to find code that works on all platforms. We have only a couple of places where we were forced to resort to this kind of thing: if wx.Platform == __WXGTK__: do X elif wx.Platform == __WXMAC__: do Y etc. Thank you for trying out though. You're welcome. VirtualBox helped. bye Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: wx MenuItem - icon is missing
On Jul 5, 2011, at 4:02 AM, Laszlo Nagy wrote: def onPopupMenu(self,evt): menu = wx.Menu() for title,bitmap in self.getPopupMenuItems(): item = wx.MenuItem(None,-1,title) if bitmap: item.SetBitmap(bitmap) menu.AppendItem(item) menu.Bind(wx.EVT_MENU,self.onPopupMenuItemSelected,item) self.PopupMenu( menu, evt.GetPoint()) menu.Destroy() I have read somewhere that under GTK, I have to assign the bitmap before Append-ing the MenuItem to the Menu. So did I, but it doesn't work. Menu item icons are not showing up in Ubuntu. On Windows 7, everything is fine. What am I doing wrong? System: Ubuntu 11 amd64 Python: 2.7.1+ wx.__version__ '2.8.11.0' Hi Laszlo, Two suggestions -- 1. Post a complete example that demonstrates the problem so that we don't have to dummy up a wx app ourselves to try your code. 2. Ask on the wxPython mailing list. Good luck Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: wx MenuItem - icon is missing
On Jul 5, 2011, at 3:32 PM, Laszlo Nagy wrote: 1. Post a complete example that demonstrates the problem so that we don't have to dummy up a wx app ourselves to try your code. [code sample snipped] Under windows, this displays the icon for the popup menu item. Under GTK it doesn't and there is no error message, no exception. I get different results than you. Under Ubuntu 9.04 w with wx 2.8.9.1, when I right click I see a menu item called test with little icon of a calculator or something. Under OS X 10.6 with wx 2.8.12.0 and Win XP with wx 2.8.10.1, when I right click I get this -- Traceback (most recent call last): File x.py, line 46, in onPopupMenu item = wx.MenuItem(None,-1,uTest) File /usr/local/lib/wxPython-unicode-2.8.12.0/lib/python2.6/site-packages/wx-2.8-mac-unicode/wx/_core.py, line 11481, in __init__ _core_.MenuItem_swiginit(self,_core_.new_MenuItem(*args, **kwargs)) wx._core.PyAssertionError: C++ assertion parentMenu != NULL failed at /BUILD/wxPython-src-2.8.12.0/src/common/menucmn.cpp(389) in wxMenuItemBase(): menuitem should have a menu Hope this helps more than it confuses. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: unzip problem
On Jun 24, 2011, at 10:55 AM, Ahmed, Shakir wrote:
Hi,
I am getting following error message while unziping a .zip file. Any
help or idea is highly appreciated.
Error message
Traceback (most recent call last):
File C:\Zip_Process\py\test2_new.py, line 15, in module
outfile.write(z.read(name))
IOError: (22, 'Invalid argument')
Start debugging with these two steps --
1) Add this just after for name in z.namelist():
print name
That way you can tell which file is failing.
2) You can't tell whether you're getting an error on the write or the read
because you've got two statements combined into one line. Change this --
outfile.write(z.read(name))
to this --
data = z.read(name)
outfile.write(data)
Good luck
Philip
The script is here:
*
fh = open('T:\\test\\*.zip', 'rb')
z = zipfile.ZipFile(fh)
for name in z.namelist():
outfile = open(name, 'wb')
outfile.write(z.read(name))
print z
print outfile
outfile.close()
fh.close()
winmail.dat--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
Re: search through this list's email archives
On Jun 23, 2011, at 12:11 PM, Cathy James wrote: Dear All, I looked through this forum's archives, but I can't find a way to search for a topic through the archive. Am I missing something? http://www.google.com/search?q=site%3Amail.python.org%2Fpipermail%2Fpython-list%2F+++banana -- http://mail.python.org/mailman/listinfo/python-list
Re: dummy, underscore and unused local variables
On Jun 13, 2011, at 11:37 AM, Tim Johnson wrote: NOTE: I see much on google regarding unused local variables, however, doing a search for 'python _' hasn't proved fruitful. Yes, Google's not good for searching punctuation. But 'python underscore dummy OR unused' might work better. On a related note: from the python interpreter if I do help(_) I get Help on bool object: class bool(int) | bool(x) - bool .. I'd welcome comments on this as well. In the Python interpreter, _ gives you the results of the last expression. When you first start the interpreter, _ is undefined. $ python help(_) Traceback (most recent call last): File stdin, line 1, in module NameError: name '_' is not defined True True help(_) Help on bool object: class bool(int) | bool(x) - bool In your case when you asked for help(_), the last object you used must have been a bool. :) I expect to be edified is so many ways, some of them unexpected. That's the nice thing about this list! Hope this helps Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Validating string for FDQN
On Jun 6, 2011, at 8:40 PM, Eric wrote: Hello, Is there a library or regex that can determine if a string is a fqdn (fully qualified domain name)? I'm writing a script that needs to add a defined domain to the end of a hostname if it isn't already a fqdn and doesn't contain the defined domain. The ones here served me very well: http://pyxml.cvs.sourceforge.net/viewvc/pyxml/xml/xml/Uri.py?revision=1.1view=markup bye Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about SQLite + Python and twitter
On May 25, 2011, at 2:17 PM, Jayme Proni Filho wrote: Helo guys, I'm building a local application for twitter for my brother's store. I'm in the beginning and I have some newbie problems, so: I create a table called tb_messages with int auto increment and varchar(140) fields; I did three SQL funcionts, insert_tweet, delete_tweet, select_tweet select_tweet is use for getting messages for sending them to twitter; My problem is: How can i make my select_tweet works at the same time that insert or delete funcions. I just got to work when I stop select function. I would like to do my app works all the time. Hi Jayme, You need to provide a lot more information for us to be able to help you. Some suggestions -- http://www.istf.com.br/perguntas/#beprecise bye Philip -- http://mail.python.org/mailman/listinfo/python-list
[issue8426] multiprocessing.Queue fails to get() very large objects
Changes by Philip Semanchuk osvens...@users.sourceforge.net: -- nosy: +osvenskan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8426 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: checking if a list is empty
On May 6, 2011, at 5:57 PM, scattered wrote: On May 6, 2:36 am, Jabba Laci jabba.l...@gmail.com wrote: Hi, If I want to check if a list is empty, which is the more pythonic way? li = [] (1) if len(li) == 0: ... or (2) if not li: ... Thanks, Laszlo is there any problem with (3) if li == []: ? What if it's not a list but a tuple or a numpy array? Often I just want to iterate through an element's items and I don't care if it's a list, set, etc. For instance, given this function definition -- def print_items(an_iterable): if not an_iterable: print The iterable is empty else: for item in an_iterable: print item I get the output I want with all of these calls: print_items( list() ) print_items( tuple() ) print_items( set() ) print_items( numpy.array([]) ) Given this slightly different definition, only the first call gives me the output I expect: def print_items(an_iterable): if an_iterable == []: print The iterable is empty else: for item in an_iterable: print item I find I use the the former style (if not an_iterable) almost exclusively. bye Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: ElementTree XML parsing problem
On Apr 27, 2011, at 2:26 PM, Mike wrote: I'm using ElementTree to parse an XML file, but it stops at the second record (id = 002), which contains a non-standard ascii character, ä. Here's the XML: ?xml version=1.0? snapshot time=Mon Apr 25 08:47:23 PDT 2011 records record id=001 education=High School employment=7 yrs / record id=002 education=Universität Bremen employment=3 years / record id=003 education=River College employment=5 yrs / /records /snapshot The complaint offered up by the parser is Unexpected error opening simple_fail.xml: not well-formed (invalid token): line 5, column 40 You've gotten a number of good observations suggestions already. I would add that if you're saving your XML file from a text editor, make sure you're saving it as UTF-8 and not ISO-8859-1 or Win-1252. bye Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Terrible FPU performance
On Apr 26, 2011, at 1:34 PM, Mihai Badoiu wrote: Already did. They suggested the python list, because the asm generated code is really correct and the problem might be with the python running on top. Does the same timing in consistency appear when you use pure Python? bye Philip On Tue, Apr 26, 2011 at 1:04 PM, Chris Colbert sccolb...@gmail.com wrote: On Tue, Apr 26, 2011 at 8:40 AM, Mihai Badoiu mbad...@gmail.com wrote: Hi, I have terrible performance for multiplication when one number gets very close to zero. I'm using cython by writing the following code: You should ask this question on the Cython users mailing list. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: De-tupleizing a list
On Apr 25, 2011, at 11:28 PM, Gnarlodious wrote:
I have an SQLite query that returns a list of tuples:
[('0A',), ('1B',), ('2C',), ('3D',),...
What is the most Pythonic way to loop through the list returning a
list like this?:
['0A', '1B', '2C', '3D',...
This works for me -
result = [('0A',), ('1B',), ('2C',), ('3D',), ]
result = [row[0] for row in result]
Cheers
Philip
--
http://mail.python.org/mailman/listinfo/python-list
Re: renaming files in OS X
On Apr 20, 2011, at 10:02 AM, jyoun...@kc.rr.com jyoun...@kc.rr.com wrote:
Hello,
I'm considering using os.rename or shutil for renaming
files on OS X (Snow Leopard). However, I've read that
shutil doesn't copy the resource fork or metadata for
the files on OS X. I'm not sure about os.rename though.
I need to keep the resource fork and metadata. Is it
better if I just use os.system('mv …') or is os.rename
safe to use?
Hi Jay,
I don't know if os.rename() does what you want, but why don't you try a simple
test and find out? Surely an empirical test is at least as useful as an answer
from someone like me who may or may not know what he's talking about. =)
The OS X command xattr shows whether or not a file has extended attributes,
which are what I think you're referring to when you say metadata. xattr is
written (badly) in Python; on my system it lives in /usr/bin/xattr-2.6
You might also find this helpful:
http://jonsview.com/mac-os-x-resource-forks
Hope this helps
Philip
--
http://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing, shared memory vs. pickled copies
On Apr 7, 2011, at 3:41 AM, John Ladasky wrote: Following up to my own post... On Apr 6, 11:40 pm, John Ladasky lada...@my-deja.com wrote: What's up with that? Apparently, what's up is that I will need to implement a third method in my ndarray subclass -- namely, __reduce__. http://www.mail-archive.com/numpy-discussion@scipy.org/msg02446.html I'm burned out for tonight, I'll attempt to grasp what __reduce__ does tomorrow. Again, I'm going to point out that, given the extent that multiprocessing depends upon pickling, pickling should be made easier. This is Python, for goodness' sake! I'm still surprised at the hoops I've got to jump through. Hi John, My own experience has been that when I reach a surprising level of hoop jumping, it usually means there's an easier path somewhere else that I'm neglecting. But if pickling subclasses of numpy.ndarray objects is what you really feel you need to do, then yes, I think asking on the numpy list is the best idea. Good luck Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing
On Apr 7, 2011, at 8:57 PM, Kerensa McElroy wrote:
Hi,
thanks for your response.
I checked out multiprocessing.value, however from what I can make out, it
works with object of only a very limited type. Is there a way to do this for
more complex objects? (In reality, my object is a large multi-dimensional
numpy array).
Elsa,
Are you following the current thread in this list which is talking about
sharing numpy arrays via multiprocessing?
http://mail.python.org/pipermail/python-list/2011-April/1269173.html
Date: Wed, 6 Apr 2011 22:20:06 -0700
Subject: Re: multiprocessing
From: drsali...@gmail.com
To: kerensael...@hotmail.com
CC: python-list@python.org
On Wed, Apr 6, 2011 at 9:06 PM, elsa kerensael...@hotmail.com wrote:
Hi guys,
I want to try out some pooling of processors, but I'm not sure if it
is possible to do what I want to do. Basically, I want to have a
global object, that is updated during the execution of a function, and
I want to be able to run this function several times on parallel
processors. The order in which the function runs doesn't matter, and
the value of the object doesn't matter to the function, but I do want
the processors to take turns 'nicely' when updating the object, so
there are no collisions. Here is an extremely simplified and trivial
example of what I have in mind:
from multiprocessing import Pool
import random
p=Pool(4)
myDict={}
def update(value):
global myDict
index=random.random()
myDict[index]+=value
total=1000
p.map(update,range(total))
After, I would also like to be able to use several processors to
access the global object (but not modify it). Again, order doesn't
matter:
p1=Pool(4)
def getValues(index):
global myDict
print myDict[index]
p1.map(getValues,keys.myDict)
Is there a way to do this
This should give you a synchronized wrapper around an object in shared memory:
http://docs.python.org/library/multiprocessing.html#multiprocessing.Value
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing, shared memory vs. pickled copies
On Apr 5, 2011, at 12:58 PM, John Ladasky wrote: Hi Philip, Thanks for the reply. On Apr 4, 4:34 pm, Philip Semanchuk phi...@semanchuk.com wrote: So if you're going to use multiprocessing, you're going to use pickle, and you need pickleable objects. OK, that's good to know. But as Dan Stromberg pointed out, there are some pickle-free ways to communicate between processes using multiprocessing. This leads straight into my second question. I THINK, without knowing for sure, that most user classes would pickle correctly by simply iterating through __dict__. So, why isn't this the default behavior for Python? Was the assumption that programmers would only want to pickle built-in classes? One can pickle user-defined classes: class Foo(object): ... pass ... import pickle foo_instance = Foo() pickle.dumps(foo_instance) 'ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.' And as Robert Kern pointed out, numpy arrays are also pickle-able. import numpy pickle.dumps(numpy.zeros(3)) cnumpy.core.multiarray\n_reconstruct\np0\n(cnumpy\nndarray\np1\n(I0\ntp2\nS'b'\np3\ntp4\nRp5\n(I1\n(I3\ntp6\ncnumpy\ndtype\np7\n(S'f8'\np8\nI0\nI1\ntp9\nRp10\n(I3\nS''\np11\nNNNI-1\nI-1\nI0\ntp12\nbI00\nS'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'\np13\ntp14\nb. As a side note, you should always use new style classes, particularly since you're exploring the details of Python class construction. New is a bit a of misnomer now, as new style classes were introduced in Python 2.2. They have been the status quo in Python 2.x for a while now and are the only choice in Python 3.x. Subclassing object gives you a new style class: class Foo(object): Not subclassing object (as you did in your example) gives you an old style class: class Foo: Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing, shared memory vs. pickled copies
On Apr 4, 2011, at 4:20 PM, John Ladasky wrote: I have been playing with multiprocessing for a while now, and I have some familiarity with Pool. Apparently, arguments passed to a Pool subprocess must be able to be pickled. Hi John, multiprocessing's use of pickle is not limited to Pool. For instance, objects put into a multiprocessing.Queue are also pickled, as are the args to a multiprocessing.Process. So if you're going to use multiprocessing, you're going to use pickle, and you need pickleable objects. Pickling is still a pretty vague progress to me, but I can see that you have to write custom __reduce__ and __setstate__ methods for your objects. Well, that's only if one's objects don't support pickle by default. A lot of classes do without any need for custom __reduce__ and __setstate__ methods. Since you're apparently not too familiar with pickle, I don't want you to get the false impression that it's a lot of trouble. I've used pickle a number of times and never had to write custom methods for it. Now, I don't know that I actually HAVE to pass my neural network and input data as copies -- they're both READ-ONLY objects for the duration of an evaluate function (which can go on for quite a while). So, I have also started to investigate shared-memory approaches. I don't know how a shared-memory object is referenced by a subprocess yet, but presumably you pass a reference to the object, rather than the whole object. Also, it appears that subprocesses also acquire a temporary lock over a shared memory object, and thus one process may well spend time waiting for another (individual CPU caches may sidestep this problem?) Anyway, an implementation of a shared-memory ndarray is here: There's no standard shared memory implementation for Python. The mmap module is as close as you get. I wrote support the posix_ipc and sysv_ipc modules which give you IPC primitives (shared memory and semaphores) in Python. They work well (IMHO) but they're *nix-only and much lower level than multiprocessing. If multiprocessing is like a kitchen well stocked with appliances, posix_ipc (and sysc_ipc) is like a box of sharp knives. Note that mmap and my IPC modules don't expose Python objects. They expose raw bytes in memory. YOu're still going to have to jump through some hoops (...like pickle) to turn your Python objects into a bytestream and vice versa. What might be easier than fooling around with boxes of sharp knives is to convert your ndarray objects to Python lists. Lists are pickle-friendly and easy to turn back into ndarray objects once they've crossed the pickle boundary. When should one pickle and copy? When to implement an object in shared memory? Why is pickling apparently such a non-trivial process anyway? And, given that multi-core CPU's are apparently here to stay, should it be so difficult to make use of them? My answers to these questions: 1) Depends 2) In Python, almost never unless you're using a nice wrapper like shmarray.py 3) I don't think it's non-trivial =) 4) No, definitely not. Python will only get better at working with multiple cores/CPUs, but there's plenty of room for improvement on the status quo. Hope this helps Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: PyThreadState_Swap crash
On Apr 4, 2011, at 9:08 AM, Wiktor Adamski wrote: I have 2 threads in C code using python 2.5.2. First thread creates new interpreter (i need several interpreters but those 2 threads use only one) like that: PyEval_AcquireLock(); threadState = Py_NewInterpreter(); PyThreadState_Swap(threadState); // calling python API PyThreadState_Swap(NULL); PyEval_ReleaseLock(); Second thread uses interpreter created in first thread: PyEval_AcquireLock(); PyThreadState_Swap(threadState); and sometimes PyThreadState_Swap crashes in debug build (PyGILState_GetThisThreadState() returns garbage). In release build that code doesn't run and so far no other problem was found. I call PyEval_InitThreads() at the begining of program and every PyEval_AcquireLock() has PyEval_ReleaseLock(). Am I allowed to use the same threadState in different threads? If I am, is there another problem in my code? Or maybe it's a bug in python - acording to documentation Python still supports the creation of additional interpreters (using Py_NewInterpreter()), but mixing multiple interpreters and the PyGILState_*() API is unsupported. - I don't use PyGILState_ but it's used internally in PyThreadState_Swap(). I also don't use PyEval_RestoreThread() - comment sugests that crashing code is present because possibility of calling from PyEval_RestoreThread(). Hi Wiktor, I'm sorry I don't have a solution or even a suggestion for you. I just wanted to point out that PyEval_AcquireLock() and PyEval_ReleaseLock() were recently deprecated: http://bugs.python.org/issue10913 Obviously they'll be around for quite a while longer but given the ominous-but-vague warning in issue10913's description, you might want to stay away from them. It's frustrating for me because I've got code I can't get to work without them. Good luck Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing, shared memory vs. pickled copies
On Apr 4, 2011, at 9:03 PM, Dan Stromberg wrote: On Mon, Apr 4, 2011 at 4:34 PM, Philip Semanchuk phi...@semanchuk.comwrote: So if you're going to use multiprocessing, you're going to use pickle, and you need pickleable objects. http://docs.python.org/library/multiprocessing.html#sharing-state-between-processes Thank you, Dan. My reading comprehension skills need work. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: calling 64 bit routines from 32 bit matlab on Mac OS X
On Mar 15, 2011, at 11:58 AM, Danny Shevitz wrote: Howdy, I have run into an issue that I am not sure how to deal with, and would appreciate any insight anyone could offer. I am running on Mac OS X 10.5 and have a reasonably large tool chain including python, PyQt, Numpy... If I do a which python, I get Mach-O executable i386. I need to call some commercial 3rd party C extension code that is 64 bit. Am I just out of luck or is there something that I can do? Depends on how desperate you are. You could install 64-bit Python alongside the 32-bit version, call with the 64-bit C DLL from 64-bit Python using ctypes, and then communicate between the 32- and 64-bit Pythons via pickled objects sent over an interprocess pipe. That solution has a Rube Goldberg-esque charm but not much else to recommend it. I hope you can find something better. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Is there any python library that parse c++ source code statically
On Mar 13, 2011, at 11:46 AM, Stefan Behnel wrote: Francesco Bochicchio, 13.03.2011 10:37: On 13 Mar, 10:14, kuangyekuangye19840...@gmail.com wrote: Hi, all. I need to generate other programming language source code from C++ source code for a project. To achieve this, the first step is to understand the c++ source code at least in formally. Thus is there any library to parse the C++ source code statically. So I can developer on this library. Since the C++ source code is rather simple and regular. I think i can generate other language representation from C++ source code. The problem is that C++ is a beast of a language and is not easy to find full parsers for it. I've never done it, but sometime I researched possible ways to do it. The best idea I could come with is doing it in 2 steps: - using gcc-xml ( http://www.gccxml.org/HTML/Index.html ) to generate an xml representation of the code - using one of the many xml library for python to read the xml equivalent of the code and then generate the equivalent code in other languages ( where you could use a template engine, but I found that the python built-in string formatting libraries are quite up to the task ). I also heard that clang is supposed to the quite useful for this kind of undertaking. I was just discussing this with some folks here at PyCon. Clang has a library interface (libclang): http://clang.llvm.org/doxygen/group__CINDEX.html There's Python bindings for it; I'm sure the author would like some company =) https://bitbucket.org/binet/py-clang/ Cheers P -- http://mail.python.org/mailman/listinfo/python-list
Re: Do you monitor your Python packages in inux distributions?
On Mar 12, 2011, at 2:26 PM, s...@pobox.com wrote: I'm one of the SpamBayes developers and in a half-assed way try to keep track of SB dribbles on the net via a saved Google search. About a month ago I got a hit on an Ubuntu bug tracker about a SpamBayes bug. As it turns out, Ubuntu distributes an outdated (read: no longer maintained) version of SpamBayes. The bug had been fixed over three years ago in the current version. Had I known this I could probably have saved them some trouble, at least by suggesting that they upgrade. I have a question for you people who develop and maintain Python-based packages. How closely, if at all, do you monitor the bug trackers of Linux distributions (or Linux-like packaging systems like MacPorts) for activity related to your packages? How do you encourage such projects to push bug reports and/or fixes upstream to you? What tools are out there to discover which Linux distributions have SpamBayes packages? (I know about rpmfind.net, but there must be other similar sites by now.) Hi Skip, I use google alerts to track where my packages posix_ipc and sysv_ipc get mentioned, and they have been turned into packages for Fedora and I think one other distro the name of which escapes me at the moment. At first I was really pleased to see them made into distro-specific packages because I'm too lazy to do it myself. But then I realized the same side effect that you described -- the versions distributed via my Web site have moved on and added bug fixes and major features like Python 3 support, while the distro-specific packages are frozen in time. I guess via my Google alerts I would learn if a bug was filed against one of my outdated packages. I only get 1-2 alerts per day, so they're easy to keep track of. If my packages were more popular, I might get so many alerts I'd just stop reading them. So far I've never seen a distro-specific bug reported against one of my packages. All bugs have been reported directly to me. I hope that continues to be the case because I don't have a good solution to the problems you mentioned. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing module in async db query
On Mar 9, 2011, at 10:22 AM, Sheng wrote: Hi Philip, multiprocessing.Queue is used to transfer data between processes, how it could be helpful for solving my problem? Thanks! I misunderstood -- I thought transferring data between processes *was* your problem. If both of your functions are in the same process, I don't understand how multiprocessing figures into it at all. If you want a function to start returning results before that function completes, and you want those results to be processed by other code *in the same process*, then you'll have to use threads. A Queue object for threads exists in the standard library too. You might find that useful. HTH Philip On Mar 8, 6:34 pm, Philip Semanchuk phi...@semanchuk.com wrote: On Mar 8, 2011, at 3:25 PM, Sheng wrote: This looks like a tornado problem, but trust me, it is almost all about the mechanism of multiprocessing module. [snip] So the workflow is like this, get() -- fork a subprocess to process the query request in async_func() - when async_func() returns, callback_func uses the return result of async_func as the input argument, and send the query result to the client. So the problem is the the query result as the result of sql_command might be too big to store them all in the memory, which in our case is stored in the variable data. Can I send return from the async method early, say immediately after the query returns with the first result set, then stream the results to the browser. In other words, can async_func somehow notify callback_func to prepare receiving the data before async_func actually returns? Hi Sheng, Have you looked at multiprocessing.Queue objects? HTH Philip -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing module in async db query
On Mar 8, 2011, at 3:25 PM, Sheng wrote: This looks like a tornado problem, but trust me, it is almost all about the mechanism of multiprocessing module. [snip] So the workflow is like this, get() -- fork a subprocess to process the query request in async_func() - when async_func() returns, callback_func uses the return result of async_func as the input argument, and send the query result to the client. So the problem is the the query result as the result of sql_command might be too big to store them all in the memory, which in our case is stored in the variable data. Can I send return from the async method early, say immediately after the query returns with the first result set, then stream the results to the browser. In other words, can async_func somehow notify callback_func to prepare receiving the data before async_func actually returns? Hi Sheng, Have you looked at multiprocessing.Queue objects? HTH Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: questions about multiprocessing
On Mar 4, 2011, at 11:08 PM, Vincent Ren wrote: Hello, everyone, recently I am trying to learn python's multiprocessing, but I got confused as a beginner. If I run the code below: from multiprocessing import Pool import urllib2 otasks = [ 'http://www.php.net' 'http://www.python.org' 'http://www.perl.org' 'http://www.gnu.org' ] def f(url): return urllib2.urlopen(url).read() pool = Pool(processes = 2) print pool.map(f, tasks) Hi Vincent, I don't think that's the code you're running, because that code won't run. Here's what I get when I run the code you gave us: Traceback (most recent call last): File x.py, line 14, in module print pool.map(f, tasks) NameError: name 'tasks' is not defined When I change the name of otasks to tasks, I get the nonnumeric port error that you reported. Me, I would debug it by adding a print statement to f(): def f(url): print url return urllib2.urlopen(url).read() Your problem isn't related to multiprocessing. Good luck Philip I'll receive this message: Traceback (most recent call last): File stdin, line 14, in module File /usr/lib/python2.6/multiprocessing/pool.py, line 148, in map return self.map_async(func, iterable, chunksize).get() File /usr/lib/python2.6/multiprocessing/pool.py, line 422, in get raise self._value httplib.InvalidURL: nonnumeric port: '' I run Python 2.6 on Ubuntu 10.10 Regards Vincent -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Checking against NULL will be eliminated?
On Mar 2, 2011, at 9:21 AM, Stefan Behnel wrote: Claudiu Popa, 02.03.2011 14:51: Hello Python-list, I don't know how to call it, but the following Python 3.2 code seems to raise a FutureWarning. def func(root=None): nonlocal arg if root: arg += 1 The warning is FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead. Why is the reason for this idiom to be changed? Let me guess - this is using ElementTree, right? It's not the idiom itself that changes, it's the Element class in ElementTree that will likely change its behaviour in later versions. Fix: do as it says. And it's documented, although you might have a hard time finding it. See the Caution at the end of this section of documentation: http://docs.python.org/py3k/library/xml.etree.elementtree.html#element-objects I wish this behavior had been changed for Python 3.x. That warning has been in the ElementTree doc since before it became part of the standard lib, so it's not a new idea. Python 3.x seems like it would have been an ideal time to make the change. Oh well. bye P -- http://mail.python.org/mailman/listinfo/python-list
Re: LDFLAGS problem
On Feb 21, 2011, at 12:56 PM, Robin Becker wrote:
After installing python 2.7.1 on a Freebsd 8.0 system with the normal
configure make dance
./configure --prefix=$HOME/PYTHON --enable-unicode=ucs2
make
make install
I find that when I build extensions PIL, MySQLdb I'm getting errors related
to a dangling ${LDFLAGS}
eg MySQLdb
running build_ext
building '_mysql' extension
creating build/temp.freebsd-7.0-RELEASE-i386-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -fPIC -Dversion_info=(1,2,2,'final',0)
-D__version__=1.2.2 -I/usr/local/include/mysql
-I/home/rptlab/PYTHON/include/python2.7 -c _mysql.c -o
build/temp.freebsd-7.0-RELEASE-i386-2.7/_mysql.o -fno-strict-aliasing -pipe
gcc -pthread -shared ${LDFLAGS}
build/temp.freebsd-7.0-RELEASE-i386-2.7/_mysql.o -L/usr/local/lib/mysql
-lmysqlclient_r -lz -lcrypt -lm -o
build/lib.freebsd-7.0-RELEASE-i386-2.7/_mysql.so
gcc: ${LDFLAGS}: No such file or directory
error: command 'gcc' failed with exit status 1
where should I be looking to fix this problem?
It's been a while since I built anything on FreeBSD, but one thing that jumps
out at me is that you say you're building on 8.0 but the build output you gave
us mentions 7.0. That doesn't sound right at all.
Are you using ports?
bye
Philip
--
http://mail.python.org/mailman/listinfo/python-list
Re: help with multiprocessing pool
On Jan 25, 2011, at 8:19 PM, Craig Yoshioka wrote:
Hi all,
I could really use some help with a problem I'm having.
Hiya Craig,
I don't know if I can help, but it's really difficult to do without a full
working example.
Also, your code has several OS X-isms in it so I guess that's the platform
you're on. But in case you're on Windows, note that that platform requires some
extra care when using multiprocessing:
http://docs.python.org/library/multiprocessing.html#windows
Good luck
Philip
I wrote a function that can take a pattern of actions and it apply it to the
filesystem.
It takes a list of starting paths, and a pattern like this:
pattern = {
InGlob('Test/**'):{
MatchRemove('DS_Store'):[],
NoMatchAdd('(alhpaID_)|(DS_Store)','warnings'):[],
MatchAdd('alphaID_','alpha_found'):[],
InDir('alphaID_'):{
NoMatchAdd('(betaID_)|(DS_Store)','warnings'):[],
InDir('betaID_'):{
NoMatchAdd('(gammaID_)|(DS_Store)','warnings'):[],
MatchAdd('gammaID_','gamma_found'):[]
so if you run evalFSPattern(['Volumes/**'],pattern) it'll return a dictionary
where:
dict['gamma_found'] = [list of paths that matched] (i.e.
'/Volumes/HD1/Test/alphaID_3382/betaID_38824/gammaID_848384')
dict['warning'] = [list of paths that failed to match] (ie.
'/Volumes/HD1/Test/alphaID_3382/gammaID_47383')
Since some of these volumes are on network shares I also wanted to
parallelize this so that it would not block on IO. I started the
parallelization by using multiprocessing.Pool and got it to work if I ran the
fsparser from the interpreter. It ran in *much* less time and produced
correct output that matched the non-parallelized version. The problem begins
if I then try to use the parallelized function from within the code.
For example I wrote a class whose instances are created around valid FS
paths, that are cached to reduce expensive FS lookups.
class Experiment(object):
SlidePaths = None
@classmethod
def getSlidePaths(cls):
if cls.SlidePaths == None:
cls.SlidePaths = fsparser(['/Volumes/**'],pattern)
return cls.SlidePaths
@classmethod
def lookupPathWithGammaID(cls,id):
paths = cls.getSlidePaths()
...
return paths[selected]
@classmethod
def fromGamaID(cls,id):
path = cls.lookupPathWithGammaID(id)
return cls(path)
def __init__(self,path)
self.Path = path
...
...
If I do the following from the interpreter it works:
from experiment import Experiment
expt = Experiment.fromGammaID(10102)
but if I write a script called test.py:
from experiment import Experiment
expt1 = Experiment.fromGammaID(10102)
expt2 = Experiment.fromGammaID(10103)
comparison = expt1.compareTo(expt2)
it fails, if I try to import it or run it from bash prompt:
from test import comparison (hangs forever)
$ python test.py (hangs forever)
I would really like some help trying to figure this out... I thought it
should work easily since all the spawned processes don't share data or state
(their results are merged in the main thread). The classes used in the
pattern are also simple python objects (use python primitives).
These are the main functions:
def mapAction(pool,paths,action):
merge = {'next':[]}
for result in pool.map(action,paths):
if result == None:
continue
merge = mergeDicts(merge,result)
return merge
def mergeDicts(d1,d2):
for key in d2:
if key not in d1:
d1[key] = d2[key]
else:
d1[key] += d2[key]
return d1
def evalFSPattern(paths,pattern):
pool = Pool(10)
results = {}
for action in pattern:
tomerge1 = mapAction(pool,paths,action)
tomerge2 = evalFSPattern(tomerge1['next'],pattern[action])
del tomerge1['next']
results = mergeDicts(results,tomerge1)
results = mergeDicts(results,tomerge2)
return results
the classes used in the pattern (InGlob,NoMatchAdd,etc.) are callable classes
that take a single parameter (a path) and return a dict result or None which
makes them trivial to adapt to Pool.map.
Note if I change the mapAction function to:
def mapAction(pool,paths,action):
merge = {'next':[]}
for path in paths:
result = action(path)
if result == None:
continue
merge = mergeDicts(merge,result)
return merge
everything works just fine.
Thanks.
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
Re: help with multiprocessing pool
On Jan 27, 2011, at 1:12 PM, Craig Yoshioka wrote:
The code will be multi-platform. The OSXisms are there as an example, though
I am developing on OS X machine.
I've distilled my problem down to a simpler case, so hopefully that'll help
troubleshoot.
I have 2 files:
test.py:
--
from multiprocessing import Pool
def square(x):
return x*x
def squares(numbers):
pool = Pool(12)
return pool.map(square,numbers)
test2.py:
--
from test import squares
maxvalues = squares(range(3))
print maxvalues
Now if I import squares into the interactive interpreter:
from test import squares
print squares(range(3))
I get the correct result, but if I try to import maxvalues from test2 the
interactive interpreter python hangs.
if I run the script from bash, though, it seems to run fine.
The short, complete example is much more useful, but it sounds like it
demonstrates a different problem than you first described. Your first posting
said that your code worked in the interpreter but failed when run from the
command line. This code has the opposite problem. Correct?
I think it might have something to do with this note in the docs, though I am
not sure how to use this information to fix my problem:
Note: Functionality within this package requires that the __main__ method be
importable by the children. This is covered inProgramming guidelines however
it is worth pointing out here. This means that some examples, such as
themultiprocessing.Pool examples will not work in the interactive interpreter.
I suspect this is the problem with the demo above. Your original code ran fine
in the interpreter, though, correct?
bye
Philip
On Jan 27, 2011, at 6:39 AM, Philip Semanchuk wrote:
On Jan 25, 2011, at 8:19 PM, Craig Yoshioka wrote:
Hi all,
I could really use some help with a problem I'm having.
Hiya Craig,
I don't know if I can help, but it's really difficult to do without a full
working example.
Also, your code has several OS X-isms in it so I guess that's the platform
you're on. But in case you're on Windows, note that that platform requires
some extra care when using multiprocessing:
http://docs.python.org/library/multiprocessing.html#windows
Good luck
Philip
I wrote a function that can take a pattern of actions and it apply it to
the filesystem.
It takes a list of starting paths, and a pattern like this:
pattern = {
InGlob('Test/**'):{
MatchRemove('DS_Store'):[],
NoMatchAdd('(alhpaID_)|(DS_Store)','warnings'):[],
MatchAdd('alphaID_','alpha_found'):[],
InDir('alphaID_'):{
NoMatchAdd('(betaID_)|(DS_Store)','warnings'):[],
InDir('betaID_'):{
NoMatchAdd('(gammaID_)|(DS_Store)','warnings'):[],
MatchAdd('gammaID_','gamma_found'):[]
so if you run evalFSPattern(['Volumes/**'],pattern) it'll return a
dictionary where:
dict['gamma_found'] = [list of paths that matched] (i.e.
'/Volumes/HD1/Test/alphaID_3382/betaID_38824/gammaID_848384')
dict['warning'] = [list of paths that failed to match] (ie.
'/Volumes/HD1/Test/alphaID_3382/gammaID_47383')
Since some of these volumes are on network shares I also wanted to
parallelize this so that it would not block on IO. I started the
parallelization by using multiprocessing.Pool and got it to work if I ran
the fsparser from the interpreter. It ran in *much* less time and produced
correct output that matched the non-parallelized version. The problem
begins if I then try to use the parallelized function from within the code.
For example I wrote a class whose instances are created around valid FS
paths, that are cached to reduce expensive FS lookups.
class Experiment(object):
SlidePaths = None
@classmethod
def getSlidePaths(cls):
if cls.SlidePaths == None:
cls.SlidePaths = fsparser(['/Volumes/**'],pattern)
return cls.SlidePaths
@classmethod
def lookupPathWithGammaID(cls,id):
paths = cls.getSlidePaths()
...
return paths[selected]
@classmethod
def fromGamaID(cls,id):
path = cls.lookupPathWithGammaID(id)
return cls(path)
def __init__(self,path)
self.Path = path
...
...
If I do the following from the interpreter it works:
from experiment import Experiment
expt = Experiment.fromGammaID(10102)
but if I write a script called test.py:
from experiment import Experiment
expt1 = Experiment.fromGammaID(10102)
expt2 = Experiment.fromGammaID(10103)
comparison = expt1.compareTo(expt2)
it fails, if I try to import it or run it from bash prompt:
from test
Re: examples of realistic multiprocessing usage?
On Jan 21, 2011, at 3:36 PM, Dan Stromberg wrote: On Fri, Jan 21, 2011 at 3:20 AM, Adam Skutt ask...@gmail.com wrote: On Jan 20, 11:51 pm, Albert van der Horst alb...@spenarnc.xs4all.nl wrote: This is what some people want you to believe. Arm twisting by GPL-ers when you borrow their ideas? That is really unheard of. Doesn't matter, you're still legally liable if your work is found to be derivative and lacking a fair use defense. It's not borrowing ideas that's problematic, it's proving that's all you did. For those of us with legal departments, we have no choice: if they don't believe we can prove our case, we're not using the code, period. The risk simply isn't worth it. Many legal departments have an overblown sense of risk, I'm afraid. I carefully avoid GPLed code on our BSD-licensed project not because I need fear anyone's legal department, but out of respect for the author(s) of the GPL-ed code. The way I see it, the author of GPL-ed code gives away something valuable and asks for just one thing in return: respect the license. It strikes me as very selfish to deny them the one thing they ask for. JMHO, Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: getdefaultencoding - how to change this?
On Jan 20, 2011, at 10:39 AM, Helmut Jarausch wrote: On Thu, 20 Jan 2011 14:31:09 +, Helmut Jarausch wrote: Hi, I've searched the net but didn't find the information I need. Using Python-2.7.1, I know, I can't modify defaultencoding at run time. Python even ignores export PYTHONIOENCODING=ISO8859-1 locale.getdefaultlocale()[1] returns 'ISO8859-1' still sys.stdout is using the ascii codec. How can I recompile Python (itself) to change this to iso8859-1 ? (My favourite editor cannot be told to use unicode.) Sorry for answering myself. One last trial did succeed. My locale as root differed from my locale as non-root user. After changing the root locale and recompiling Python, it works now. I'm glad that worked for you. Alternatively, it seems like you can set the default encoding in site.py which sounds easier than recompiling Python. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: getdefaultencoding - how to change this?
On Jan 20, 2011, at 11:47 AM, Robert Kern wrote: On 1/20/11 9:47 AM, Philip Semanchuk wrote: I'm glad that worked for you. Alternatively, it seems like you can set the default encoding in site.py which sounds easier than recompiling Python. Never do that. It breaks dicts and makes your code non-portable. I've never been tempted for the very non-portability reason you cite. I didn't know that it would break dicts, though. Thanks for the education. bye Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: examples of realistic multiprocessing usage?
On Jan 16, 2011, at 2:05 PM, TomF wrote: I'm trying to multiprocess my python code to take advantage of multiple cores. I've read the module docs for threading and multiprocessing, and I've done some web searches. All the examples I've found are too simple: the processes take simple inputs and compute a simple value. My problem involves lots of processes, complex data structures, and potentially lots of results. It doesn't map cleanly into a Queue, Pool, Manager or Listener/Client example from the python docs. Instead of explaining my problem and asking for design suggestions, I'll ask: is there a compendium of realistic Python multiprocessing examples somewhere? Or an open source project to look at? A colleague pointed me to this project the other day. http://gluino.com/ I grepped through the code to see that it's using multiprocessing.Listener. I didn't go any further than that because our project is BSD licensed and the license for Gluino is unclear. Until I find out whether or not its under an equally permissive license, I can't borrow ideas and/or code from it. Hope it's of some help to you, though. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Career path - where next?
On Jan 12, 2011, at 11:37 AM, Alan Harris-Reid wrote: Hi there, I wonder if any Python folk out there can help me. For many years I was a contractor developing desktop and web applications using Visual Foxpro as my main language, with Foxpro, SQL-server and Oracle as back-end databases. Unfortunately Foxpro was killed-off by Microsoft, hence my work dried-up and my last 'big' contract ended about a year ago. Since then I have taken time off programming doing house-renovation, and in the last 6 months I have been updating my programming skills by learning Python (3) with SQLite, JavaScript, HTML and CSS to a level where I can create and deploy data-based web-sites. My situation now is that I am reasonably comfortable with the above languages and am now in a position where I wish to return to employment using my new and/or existing skills (contract/permanent, full/part-time or teleworking). However, I have yet to find any UK vacancy which will accept a relative 'beginner' - they all require at least 2-3 years Python in a commercial environment. It's a catch-22 situation - it's hard to get a job without experience, but you need a job to get experience in the 1st place! I would even consider doing small projects for nothing so that I can 'get my foot in the door' (although I hope to be wise-enough to know when I am being taken advantage of!). I am also mailing CVs to agencies I think may be interested. If anyone out has ideas as to how to proceed towards achieving my goal, I would be grateful for any advice. Contributing to open source projects (your own or someone else's) will help to convince some employers that you're worth taking a look at. If nothing else it gives you a public example of the work that you can point them to. Good luck Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: What INI config file module allows lists of duplicate same-named options?
On Jan 10, 2011, at 6:05 PM, Ben Finney wrote: Thomas L. Shinnick tshin...@io.com writes: Here, I need to list multiple file/dir path pairs. A list of multiple items to be acted upon in a common way. It is a list. Simple. Except I can't find a library/pypi module with the obvious extension. What you want is incompatible with calling the result “an INI file”, because that entails the restrictions you described. I dunno about that. The INI file format isn't standardized so there aren't restrictions on what one can expect to find in an INI file other than people's expectations. I'll grant you that most INI files don't have (or expect) duplicate keys in a section, but I've seen some that do. You would be better advised to use a configuration format that can do what you want, such as YAML or JSON. Both of those have good Python support; JSON in particular has support in the standard library. I second that, and the point above (about there being no standard that governs INI files) is another reason to avoid them. Some INI file libraries expect a hash mark as a comment, some expect semicolon, some make no allowances for non-ASCII encodings, some expect UTF-8 or ISO-8859-1 or Win-1252, some only allow '=' as the key/value separator, some allow other characters. INI files are nice and simple but there's devils in those details. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Searching Python-list
On Jan 5, 2011, at 8:31 PM, Slie wrote: I was wondering if anyone could tell me how to search through the Archives otter then manually looking through each month. Do a Google search and include this term: site:mail.python.org/pipermail/python-list/ e.g. to search for banana: http://www.google.com/search?q=site:mail.python.org%2Fpipermail%2Fpython-list%2F+banana HTH Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: is py2exe still active ?
On Dec 8, 2010, at 5:09 AM, Octavian Rasnita wrote: Hi Steve, I may put some stupid questions because I am very new to Python, but... I heard about pypi/pip. Aren't all these Python libraries (like cxFreeze) provided on a central archive where we can get them and also report the bugs using a single request/issue tracker? Hi Octavian, I didn't see anyone reply to you on the list... The short answer to your question is no. PyPI stands for the Python Package Index. The key word there is Index. It's a catalog of packages but many of those packages are hosted elsewhere. The places where those packages are hosted may or may not have an issue tracker, etc. For instance, one the packages that I offer through PyPI (posix_ipc) is hosted on my personal Web site. Hope this helps Philip - Original Message - From: Steve Holden st...@holdenweb.com Newsgroups: gmane.comp.python.general To: Octavian Rasnita orasn...@gmail.com Cc: python-list@python.org Sent: Wednesday, December 08, 2010 12:56 AM Subject: Re: is py2exe still active ? Octavian: It's great that you want to let people know about bugs. Put yourself in the position of the package maintainer, however. She or he doesn't spend all day working on cxFreeze, and probably doesn't even do a Google search on cxFreeze very often. So they are unlikely to find out about this problem form your well-intentioned note. It's just possible nobody does care, as I can't find a link to an issue tracker - the best I could advise in this case would be to join the mailing list by visiting https://lists.sourceforge.net/lists/listinfo/cx-freeze-users regards Steve On 12/7/2010 6:49 PM, Octavian Rasnita wrote: This packager is also nice. If someone cares, I've discovered a small bug in it. If Python is installed on another drive than C under Windows, the cxfreeze.bat file still calls Python on the drive C and it doesn't work until it is corrected. Octavian - Original Message - From: Cbast sebastien.fri...@gmail.com Newsgroups: comp.lang.python To: python-list@python.org Sent: Tuesday, December 07, 2010 5:00 PM Subject: Re: is py2exe still active ? On Dec 7, 8:23 am, Anders Persson anders.u.pers...@gmail.com wrote: Hi! When a look att py2exe homepage it is not looking like mutch happen, as a beginner i was thinking to start with Python 3, but i like to now if py2exe will be for 3 too. Is any one have any info ? I don't have the answer about py2exe, but I'm using cxFreeze to create executables with Python 3.1, if it's what you're looking for. http://cx-freeze.sourceforge.net/ -- Steve Holden +1 571 484 6266 +1 800 494 3119 PyCon 2011 Atlanta March 9-17 http://us.pycon.org/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Backup postgresql database from python
On Dec 4, 2010, at 2:32 PM, D'Arcy J.M. Cain wrote: On Sat, 4 Dec 2010 19:12:08 + starglider develop starglider@gmail.com wrote: I need to backup a postgresql database from python withour using pg_dump! Is any way of doing that? Probably. I guess the first question is why can't you use pg_dump? That might give us a clue as to the requirements. Excellent point. No offense to the OP, but this isn't really a Python question. You could re-implement pg_dump in Python, Javascript, or any language you like and you'd have your solution. That's probably not what you were looking for though. As D'Arcy said, the first thing to establish is why you want to avoid pg_dump. Another important question is whether or not you expect the database to be in use while you're doing backups. bye Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Installing pysqlite on Win64
On Nov 20, 2010, at 12:37 PM, Navid Parvini wrote: Dear All, I want to install pysqlite on my Windows 64 bit machine. I have python 2.4.3 on it. Would you please let me know how can I do it? That is, I need to use the source file (i.e pysqlite-2.5.6.tar.gz) or there is an executable file to install. Hi Navid, I'm not a Windows guy so I really can't recommend how to proceed with the specific question you asked. But if there's any way you could upgrade the machine to Python = 2.5, your problem will go away because those versions of Python come with a built-in sqlite interface. That might be an easier approach for you. Good luck Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Unladen Swallow dead?
On Nov 17, 2010, at 5:09 PM, John Nagle wrote: On 11/17/2010 12:49 PM, John Ladasky wrote: On Nov 16, 2:30 pm, laspilorena.aspi...@gmail.com wrote: Is Unladen Swallow dead? No, it's just resting. For those who don't get that, The Monty Python reference: http://www.mtholyoke.edu/~ebarnes/python/dead-parrot.htm; A link to the source material: http://www.youtube.com/user/montypython?blend=1ob=4#p/c/6FD5A97331C1B802/0/npjOSLCR2hE -- Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Copy Protected PDFs and PIL
On Nov 11, 2010, at 3:28 PM, Brett Bowman wrote: I'm trying to parse some basic details and a thumbnail from ~12,000 PDFs for my company, but a few hundred of them are copy protected. To make matters worse, I can't seem to trap the error it causes: whenever it happens PIL throws a FATAL PDF disallows copying message and dies. An automated way to snap a picture of the PDFs would be ideal, but I'd settle for a way to skip over them without crashing my program. Any tips? What operating system? -- http://mail.python.org/mailman/listinfo/python-list
Re: Silly newbie question - Caret character (^)
On Nov 9, 2010, at 1:04 AM, Lawrence D'Oliveiro wrote: In message slrnidhcns.9m6.usenet-nos...@guild.seebs.net, Seebs wrote: On 2010-11-09, Lawrence D'Oliveiro l...@geek-central.gen.new_zealand wrote: In message mailman.697.1289067607.2218.python-l...@python.org, Dennis Lee Bieber wrote: Have you ever looked at the reference manual for Ada? Or even worse, the annotated reference. I thought annotations were supposed to clarify things; in this case they seemed to have the opposite effect... Clearly, you've never seen Schildt's infamous Annotated ANSI C Standard. There absolutely no chance (or at least no way I can imagine) that anyone could annotate a concise language like C up to the level of turgidity of the Ada spec. Hang on, is this Herb Schildt? I bought a couple of his books, back when I was trying to get to grips with C++ (I have the edition of “C++ The Complete Reference” which proudly proclaims it “Covers the New International Standard for C+”). Not as useful as I thought they would be; I ended up referring to the libstdc++ sources to clarify things. What's funny is that I went looking for a printed copy of the C standard a few years back and the advice I got was that the cheapest route was to find a used copy of Schildt's Annotated ANSI C Standard and ignore the annotations. So it serves at least one useful purpose. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Silly newbie question - Carrot character (^)
On Nov 6, 2010, at 10:45 AM, D'Arcy J.M. Cain wrote: On Sat, 6 Nov 2010 10:22:47 -0400 Philip Semanchuk phi...@semanchuk.com wrote: The tutorial isn't meant as an exhaustive lesson on every single Python feature. I agree, and I don't expect otherwise. My point was that if the tutorial doesn't mention a feature, the only other place to learn about it (on python.org) is the language ref. Some people might think the language ref is a fine place to direct newcomers to Python. I don't. I don't think that anyone was suggesting the reference as the first place to send newcomers. Hi D'Arcy, I agree, no one -- not even I -- suggested that. The tutorial is always the first stop. You send them there when they need something beyond the basics. But IMO this is still a problem. After the tutorial, then what? Someone who has read the tutorial can still be new to Python by my definition. You may feel that the current language ref is OK for newcomers. I don't, and that's my point. I think the only issue here is that operators are pretty basic and that specific thing is missing in the tutorial. It would be a mistake to write a whole new document because the tutorial is missing one thing. Better would be to propose an operators section. The issue is not just operators. As I mentioned, the tutorial doesn't cover decorators, assert, exec, ternary if, and maybe a few other things. IMO that's fine. A tutorial should introduce the basics. I'm sure we could have a fine argument about what features of Python are basic and which are advanced, but I'd rather not as long as we can agree with Steven D'Aprano's comment that The tutorial isn't meant as an exhaustive lesson on every single Python feature. I certainly agree with that. Personally, I liked using the tutorial for learning Python. It's readable. But after getting familiar with the language I wanted to move on to something more structured. I also realized that the tutorial didn't cover every aspect of the language because I saw references in code and discussions to things that weren't mentioned in the tutorial. I didn't feel like the python.org documentation provided an obvious next step, though, because I started with Python 2.3, and the language reference was still entitled for language lawyers back then. The section's name may have changed since then, but it looks like the style hasn't changed much. I believe the lack of a complete, friendly post-tutorial document to read made learning Python more difficult for me. I realize that the Python Foundation doesn't have infinite resources to work with, so maybe they'd love to create maintain a more readable language reference if they had time/money/people. I don't hear anyone talk about it, though. Lots of people talk. Action, not so much. How about you? Are you ready to start writing a new reference manual? First you suggest that writing a whole new document would be a mistake, then you encourage me to do it. =) The old open source challenge of If you don't like it, fix it is liberating but it's also a nifty way of changing the subject. Whether or not I can or will fix a problem doesn't make my criticism any more or less legitimate which (I thought) was the issue at hand. To answer your question: no, I'm not willing to start writing a new reference manual. For one thing, I don't think it's wanted judging by the response I've seen here. I see a lot of people saying what we have is fine. To put it another way, based on this small sample size survey, opinions are mixed on the current state of the documentation. The effort required to make substantial changes (e.g. create a reference manual that's sort of a marriage of the tutorial and the language spec) is large but would likely result in very little net improvement as perceived by the community as a whole. To put it a fifth way (Sir Galahad: third, sir) -- people learn differently. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Silly newbie question - Carrot character (^)
On Nov 6, 2010, at 12:33 AM, Steven D'Aprano wrote: On Fri, 05 Nov 2010 23:21:11 -0400, Philip Semanchuk wrote: Take the OP's question. How is one supposed to find out about bitwise operators in Python? AFAICT they're not mentioned in the tutorial, and neither are decorators, assert(), global, exec, the ternary if statement, etc. The tutorial isn't meant as an exhaustive lesson on every single Python feature. I agree, and I don't expect otherwise. My point was that if the tutorial doesn't mention a feature, the only other place to learn about it (on python.org) is the language ref. Some people might think the language ref is a fine place to direct newcomers to Python. I don't. It's not awful, but it's dense and unfriendly for those just starting out. There are plenty of other resources available: learning Python *starts* with the python.org tutorial (or some equivalent), it doesn't end there. Yes, I agree. That's what I said in my email too. One goes through the tutorial a few times and then...? There's not a formal document to turn to after that. There are plenty of resources -- books, mailing lists, etc. But they're 3rd party, unstructured, not maintained, etc. I realize that the Python Foundation doesn't have infinite resources to work with, so maybe they'd love to create maintain a more readable language reference if they had time/money/people. I don't hear anyone talk about it, though. bye Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Silly newbie question - Carrot character (^)
On Nov 5, 2010, at 9:43 AM, Matty Sarro wrote: Hey Everyone, Just curious - I'm working on a program which includes a calculation of a circle, and I found myself trying to use pi*radius^2, and getting errors that data types float and int are unsupported for ^. Now, I realized I was making the mistake of using '^' instead of **. I've corrected this and its now working. However, what exactly does ^ do? I know its used in regular expressions but I can't seem to find anything about using it as an operator. Sadly my google foo is failing since the character gets filtered out. As others have said, ^ is for XOR. That's buried here in the documentation: http://docs.python.org/release/2.7/reference/expressions.html#binary-bitwise-operations Not that I would have expected you to find it there since that's pretty dense. In fact, older versions of the Python doc used to describe this section as for language lawyers but I see they've changed that now. BTW the more common name for this character is caret (ka-RAY). Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Silly newbie question - Carrot character (^)
On Nov 5, 2010, at 12:43 PM, Peter Pearson wrote: On Fri, 5 Nov 2010 10:12:05 -0400, Philip Semanchuk wrote: BTW the more common name for this character is caret (ka-RAY). Yes, it's caret, but no, it's KA-rit, almost the same as carrot. It's straight from Latin, with no detour through French. This I did not know (obviously). Thanks. I knew all those years of studying French would ruin me somehow. -- http://mail.python.org/mailman/listinfo/python-list
Re: Silly newbie question - Carrot character (^)
On Nov 5, 2010, at 5:21 PM, Nobody wrote: On Fri, 05 Nov 2010 10:12:05 -0400, Philip Semanchuk wrote: As others have said, ^ is for XOR. That's buried here in the documentation: http://docs.python.org/release/2.7/reference/... Not that I would have expected you to find it there since that's pretty dense. In fact, older versions of the Python doc used to describe this section as for language lawyers but I see they've changed that now. However, it's still written for language lawyers. IMHO, the lack of a reference manual for the language itself is a major hole in Python's documentation. I agree. -- http://mail.python.org/mailman/listinfo/python-list
Re: Silly newbie question - Carrot character (^)
On Nov 5, 2010, at 6:51 PM, Seebs wrote: On 2010-11-05, Nobody nob...@nowhere.com wrote: However, it's still written for language lawyers. IMHO, the lack of a reference manual for the language itself is a major hole in Python's documentation. I'm a bit lost here. Could you highlight some of the differences between a reference manual for the language itself and something written for language lawyers? The former refers to something that programmers would use to learn the language once they've gone through the tutorial a few times. The latter is great for writing a Python parser but isn't the friendliest guide to language constructs. Take the OP's question. How is one supposed to find out about bitwise operators in Python? AFAICT they're not mentioned in the tutorial, and neither are decorators, assert(), global, exec, the ternary if statement, etc. It seems that plowing through a document written for language lawyers is the only formal way to learn about those language features, and that could be improved upon IMO. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Compiling/Installing Python 2.7 on OSX 10.6
On Nov 4, 2010, at 4:05 PM, Ned Deily wrote: In article 238cec6d-2f47-4c97-8941-e28e68089...@a9g2000pro.googlegroups.com, Jeremy jlcon...@gmail.com wrote: [...] I downloaded the source from python.org and extracted with 'tar -xzvf Python-2.7.tgz' My home space is on some network somewhere. I think the network filesystem creates the ._ at the beginning of the files. It's really quite annoying. It is and really shouldn't be happening. If I understand correctly, whoever administers your system is doing its users a disservice by putting OS X home directories on such a file system. The path names look a little suspicious, too: /home/jlconlin. What file system type are these files on? You shouldn't run into problems if you use an HFS+ file system (for instance) and extract the tarball from the command line using /usr/bin/tar. I am intentionally installing in my home directory (i.e., /home/ jlconlin) because I don't have access to /usr/local. Supposedly this is possible, and in fact common. It is common and not normally a problem. I was just noting that the path name was not the OS X default of /Users/jlconlin. That said, there are a couple of options. Either find another file system to install to or, after extracting, you may be able to delete the spurious '._' files by a judicious use of find (-name '\.\_*' perhaps), or you could probably just ignore all the compiling errors. Those aren't compile errors in the sense of C compiler errors; rather they are from one of the final install steps that produces optimized .pyc and .pyo versions of all of the standard library .py files. The ._ files aren't python files but they do end in .py so compileall mistakenly tries to bytecompile them, too. You might want to try this before running tar to see if it inhibits the ._ files: export COPYFILE_DISABLE=True I know that tells tar to ignore those files (resource forks, no?) when building a tarball. I don't know if it helps with extraction though. Good luck Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Create a GUI and EXE for a python app?
On Oct 28, 2010, at 5:43 PM, brad...@hotmail.com wrote: Thanks ill give it a try! Anyone know about the GUI then? Lots of people know about GUIs, the problem is that it's not a simple subject. There's lots of free education on the subject in the list archives. Here's my very brief summary of options -- - Tkinter - unsophisticated (-) but built into Python (+++) - wxPython - rich (+), not built into Python (-) - pyQT - rich (+), not built into Python (-), many people say it is nicer than wxPython - There are other choices but I know nothing about them. There may be license issues for you to consider. Also, my very limited experience with py2exe is that it is less likely to work the more complicated your app is. So if you start adding in wxPython or pyQT or some other GUI, py2exe might not be able to bundle your app anymore. As I intimated, there are many tradeoffs to be considered. If this is your first Python app, I'd say keep it simple and stick with Tkinter. Worry about the fancy stuff later. Good luck Philip --Original Message-- From: Chris Rebert Sender: ch...@rebertia.com To: Braden Faulkner Cc: python-list@python.org Subject: Re: Create a GUI and EXE for a python app? Sent: Oct 28, 2010 5:04 PM On Thu, Oct 28, 2010 at 1:53 PM, Braden Faulkner brad...@hotmail.com wrote: Having trouble with my mail client, so sorry if this goes through more than once. I'm worknig on a simple math program as my first application. I would like to make a cross-platform pretty GUI for it and also package it up in a EXE for distribution on Windows. What are the best and easiest ways I can do this? For the latter, py2exe: http://www.py2exe.org/ Cheers, Chris Sent wirelessly from my BlackBerry. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Python 2.7 or 3.1
On Oct 26, 2010, at 11:10 PM, Jorge Biquez wrote: Hello Christian and all . Thanks for the comments. I am newbie to Python trying to learn all the comments, that , by the way, I am very impressed of the knowledge of the people present in this list. I was wondering if you can comment more about what alternatives to use instead to MySql. My web solutions do not need all the power of a true database, I even was wondering if I couldbe able to put simple dBase files (yes, dBase files) with my web solutions. - Any comments you can do on what to use 2.7 or 3.1? ( I guess 2.7 for what I have read) - Maybe should be another subject but... Any comments on using dBase format file with Python? Hi Jorge, Python comes with SQLite baked in, meaning you don't have to install anything extra to get the full power of SQLite. Depending on what you want to do, that might be perfect for your needs. It's been part of Python since 2.5. If you need a heavy-duty database, I recommend checking out PostgreSQL. I've always found it solid and easy to use. Have fun Philip At 08:50 p.m. 26/10/2010, you wrote: Am 27.10.2010 03:38, schrieb Jorge Biquez: And what about if I only were to develop for the web? I mean web applications, Mysql, etc? It would be better to still be in 2.7? Most frameworks and database adapters at least target Python 2.6+ as their main Python version. I guess the majority has no or only experimental support for Python 3.1. The overall situation improves every week. Christian PS: I recommend against MySQL, if you need the full power or a RDBMS. Just try to combine foreign keys with database triggers and you'll see which major features are still broken in MySQL. But that's just my point of view as a power user. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Help Need in running a Python Program from terminal
On Oct 25, 2010, at 7:16 AM, Raji wrote: Greetings ! I want to analyse and debug a python program ( its a astrology application ) Downloaded the code from here http://openastro.org/?Download http://openastro.org/?Download%20for Ubuntu When i executed the main file python openastro.py from terminal i stopped with the following error Traceback (most recent call last): File openastro.py, line 90, in module TRANSLATION[LANGUAGES[i]] = gettext.translation(openastro,TDomain,languages=['en']) File /usr/lib/python2.6/gettext.py, line 484, in translation raise IOError(ENOENT, 'No translation file found for domain', domain) IOError: [Errno 2] No translation file found for domain: 'openastro' Hi Raji, Did you have a look at the documentation for the call that's failing? Here's the doc for gettext.translation(): http://docs.python.org/library/gettext.html#gettext.translation I don't know anything about gettext or openastro, but the doc says, If no .mo file is found, this function raises IOError... which is the problem you're having. It seems like what you downloaded is expecting to find a .mo file but can't. You might want to check the package instructions on openastro.org to make sure there's not more you need to do to install it. Good luck Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: get python bit version as in (32 or 64)
On Oct 19, 2010, at 5:18 PM, Vincent Davis wrote: How do I get the bit version of the installed python. In my case, osx python2.7 binary installed. I know it runs 64 bt as I can see it in activity monitor. but how do I ask python? sys.version '2.7 (r27:82508, Jul 3 2010, 21:12:11) \n[GCC 4.0.1 (Apple Inc. build 5493)]' I don't think there's an official way to do this. The canonical way appears to be to test the value of sys.maxint and see whether or not it is a 32- or 64-bit long. See here for more details: http://stackoverflow.com/questions/1405913/how-do-i-determine-if-my-python-shell-is-executing-in-32bit-or-64bit-mode Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: get python bit version as in (32 or 64)
On Oct 19, 2010, at 5:38 PM, Hexamorph wrote:
On 19.10.2010 23:18, Vincent Davis wrote:
How do I get the bit version of the installed python. In my case, osx
python2.7 binary installed. I know it runs 64 bt as I can see it in
activity monitor. but how do I ask python?
sys.version
'2.7 (r27:82508, Jul 3 2010, 21:12:11) \n[GCC 4.0.1 (Apple Inc. build
5493)]'
In [1]: import platform
In [2]: platform.architecture()
Out[2]: ('32bit', 'ELF')
In [3]:
Looks a lot better than my suggestion!
--
http://mail.python.org/mailman/listinfo/python-list
Re: Does everyone keep getting recruiting emails from google?
On Oct 14, 2010, at 11:49 AM, Daniel Fetchinson wrote: I keep getting recruiting emails from charlesngu...@google.com about working for google as an engineer. I know what you mean. Apparently Charles Nguyen doesn't realize that I already get no end of emails and phone calls from Sergei and Larry begging me to come work with them. They won't take a flat no over the phone but I can't stand another trip to California on the private jet (the Pouilly-Fuissé isn't properly chilled but it's better than the red which isn't fit for vinegar). The yacht trips are getting old too, stuck on the boat with Eric yammering on about stock options and tacking like a nervous maniac so I nearly get killed by the boom every five minutes. I'd rather be knocked unconscious into the Pacific than hear that unique opportunity speech again. FWIW, I got one email from Charles Nguyen and answered with a thanks but no thanks. I have not heard from him again. He's perhaps casting too broad a net but the email I got looked legitimately from Google, judging by the headers. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Compiling as 32bit on MacOSX
On Oct 13, 2010, at 3:54 AM, Gregory Ewing wrote: Philip Semanchuk wrote: Hi Greg, Are you talking about compiling Python itself or extensions? I've managed to get Python itself compiled as 32 bit, and that also seems to take care of extensions built using 'python setup.py ...'. I'm mainly concerned about non-Python libraries that get wrapped by the extensions, of which I've built up quite a collection over the years. Currently I'm having to keep a careful eye out when building them to make sure they don't get compiled with the wrong architecture, since gcc's natural inclination is to default to 64 bit whenever it's available. So I was wondering if there was some way of globally changing that default that doesn't rely on compiler options getting passed correctly through the many and varied layers of build technology that one comes across. But from what I've seen so far, it seems not. If CFLAGS isn't doing the trick for you, then I don't know what to suggest. Maybe some libs also need LDFLAGS='-arch i386 -arch x86_64'? FYI, the `file` command will give you information about whether or not a binary is 32-bit, 64-bit or both. $ file shlib/libreadline.6.1.dylib shlib/libreadline.6.1.dylib: Mach-O universal binary with 2 architectures shlib/libreadline.6.1.dylib (for architecture i386):Mach-O dynamically linked shared library i386 shlib/libreadline.6.1.dylib (for architecture x86_64): Mach-O 64-bit dynamically linked shared library x86_64 Good luck Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Compiling as 32bit on MacOSX
On Oct 12, 2010, at 8:29 PM, Gregory Ewing wrote: I'm getting my Python environment set up on a new Snow Leopard machine, and I'd like to compile everything in 32 bit mode for the time being, because some of the extensions I need use APIs that aren't available in 64 bit. Is there some environment variable or config setting that will make gcc compile 32 bit binaries by default? Setting CFLAGS isn't very reliable, since the build systems of some libraries don't seem to take notice of it. Hi Greg, Are you talking about compiling Python itself or extensions? -- http://mail.python.org/mailman/listinfo/python-list
Re: SQLite is quite SQL compliant
On Oct 3, 2010, at 2:21 PM, John Nagle wrote: On 10/2/2010 3:06 PM, Seebs wrote: I would agree that the word nonstandard seems to be a little strong and discouraging. sqlite is a source of joy, a small bright point of decent and functional software in a world full of misbehaving crap. While it does omit a few bits of SQL functionality, I'd call it perhaps a slightly incomplete implementation rather than a nonstandard variant. That's a bit much. What SQLite leaves out is the heavy machinery needed for a active production database being used by many processes. If you want to store a modest amount of data for one process, perhaps with a rare conflict when two programs hit the same table, SQLite is fine. But SQLite doesn't scale. That's why it's lite. Some of SQLite's good features, are achieved by rather brutal means. For example, SQLite supports atomic transactions. That's implemented by locking up all tables involved for the duration of the entire transaction. This is fine for low-volume updates, and a killer for high-volume systems. SQLite doesn't have a serious query optimizer, or partial table locking, or concurrent transaction handling, or replication. In other words, use SQLite in your desktop app to manage its data or configuration parameters. Use MySQL or Postgres for your web site. Granted, but we're talking about whether or not SQLite complies with the SQL standard, not whether it's suitable for an e-commerce Web site or running the NYSE. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: SQLite is quite SQL compliant
On Oct 2, 2010, at 6:58 PM, Tim Chase wrote: On 10/02/10 17:06, Seebs wrote: On 2010-10-02, Ravira.ravi@gmail.com wrote: The documentation of the sqlite module at http://docs.python.org/library/sqlite3.html says: ...allows accessing the database using a nonstandard variant of the SQL... I would agree that the word nonstandard seems to be a little strong and discouraging. sqlite is a source of joy, a small bright point of decent and functional software in a world full of misbehaving crap. While it does omit a few bits of SQL functionality, I'd call it perhaps a slightly incomplete implementation rather than a nonstandard variant. In my experience, it might be better phrased as non-standard (but more adherent to standards than Microsoft SQL-Server or MySQL) variant of SQL. :-) I mean really...does *any* RDBMS actually adhere to ANSI SQL? That's what I was thinking. Most of them achieve 90 - 98% and implement their own extra 10% of non-standard extensions. One just has to hope that the bits one needs are not in the missing 2-10%. I agree with the OP that the Python doc description of SQLite, while factually correct, seems a bit severe. Cheers Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: if the else short form
On Sep 29, 2010, at 7:19 AM, Tom Potts wrote: This is just a sneaky shorthand, which is fine if that's what you want, but it makes it harder to read. The reason it works is that 'fill==True' is a boolean expression, which evaluates to True or False, but if you force a True into being an integer, it will be 1, and a False will become 0. Try writing 'True == 1' on the Python interpreter to see what I mean. So this code snippet is creating a tuple with two elements, and then selecting the first if 'fill==True' is False, or 0, and selecting the second if 'fill==True' is True, or 1. As I say, this kind of coding is absolutely fine, but it makes things harder to read and doesn't really save much space. I wouldn't recommend using this kind of style yourself, at least until you're more familiar with programming in Python. Does Python make any guarantee that int(True) == 1 and int(False) == 0 will always hold, or are their values an implementation detail? Thanks Philip On 29 September 2010 11:42, Tracubik affdfsdfds...@b.com wrote: Hi all, I'm studying PyGTK tutorial and i've found this strange form: button = gtk.Button((False,, True,)[fill==True]) the label of button is True if fill==True, is False otherwise. i have googled for this form but i haven't found nothing, so can any of you pass me any reference/link to this particular if/then/else form? thanks Nico -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
how to test get_special_folder_path()?
Hi all, The documentation for get_special_folder_path() and friends says that they're available as additional built-in functions in the installation script. http://docs.python.org/distutils/builtdist.html#the-postinstallation-script Does anyone know of a way to play around with these functions outside of a post-install script? It's time-consuming to install something just to experiment with a single function. Thanks Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Parsing error for ConfigParser
On Sep 23, 2010, at 1:22 PM, Andrew Z. wrote:
Is there a way to parse RealPlayer's realplayerrc in Python? I need
to support Python 2.5 - 2.7
Example code
import urllib2
import ConfigParser
f = urllib2.urlopen('http://pastebin.com/download.php?i=N1AcUg3w')
config = ConfigParser.RawConfigParser()
config.readfp(f)
Error
Traceback (most recent call last):
File test_config_parser_real_player.py, line 6, in module
config.readfp(f)
File /usr/lib/python2.6/ConfigParser.py, line 305, in readfp
self._read(fp, filename)
File /usr/lib/python2.6/ConfigParser.py, line 510, in _read
raise e
ConfigParser.ParsingError: File contains parsing errors: ???
[line 31]: '%0aCopyright (c) 1995-2000 Macromedia, Inc. All
rights reserved.\r\n'
[line 34]: '%0a\r\n'
Hi Andrew,
Hopefully someone familiar with the topic of RealPlayer file formats will step
in, but while you're waiting here's my $.02.
Looks like this is an INI file, and since there's no defined standard for INI
files, it's hard for anyone to write a parser that is guaranteed to work with
all INI files.
In this particular case you've got a multiline entry in the INI file which
apparently ConfigParser doesn't like. One approach would be to subclass
ConfigParser and massage the lines it doesn't like into something ConfigParser
finds more palatable. Or if you're not interested in those lines, parse the
error message, delete the offending lines and re-run the INI file through
ConfigParser.
Good luck
Philip
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python in Linux - barrier to Python 3.x
On Sep 21, 2010, at 8:29 AM, Ant wrote: Hi all, I've just seen this: http://sheddingbikes.com/posts/1285063820.html Whatever you think of Zed Shaw (author of the Mongrel Ruby server and relatively recent Python convert), he has a very good point in this. I run Fedora 12 on my home computers, and find it far too much hassle to try to get Python 3 installed. Even the 2.x's are behind - IIRC think it currently uses 2.5. Don't know about Python 3 on Fedora (I use a Mac), but distrowatch.org reports that Fedora has been using Python = 2.6 since Fedora 11 which was released in June of 2009. http://distrowatch.com/table.php?distribution=fedora bye Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: Python in Linux - barrier to Python 3.x
On Sep 21, 2010, at 8:29 AM, Ant wrote: Hi all, I've just seen this: http://sheddingbikes.com/posts/1285063820.html Whatever you think of Zed Shaw (author of the Mongrel Ruby server and relatively recent Python convert), he has a very good point in this. I run Fedora 12 on my home computers, and find it far too much hassle to try to get Python 3 installed. Even the 2.x's are behind - IIRC think it currently uses 2.5. Don't know about Python 3 on Fedora (I use a Mac), but distrowatch.org reports that Fedora has been using Python = 2.6 since Fedora 11 which was released in June of 2009. http://distrowatch.com/table.php?distribution=fedora bye Philip -- http://mail.python.org/mailman/listinfo/python-list
Re: This Is International Don’t-Squawk-Like-A -Parrot Day
On Sep 19, 2010, at 7:37 AM, Tim Chase wrote: On 09/18/10 23:46, Lawrence D'Oliveiro wrote: Do your bit to help stamp out parrocy. Did you send this by mistake? It looks like a parroty-error. I think it's a bit off... What an wkward thing to say. Are you crackers? -- http://mail.python.org/mailman/listinfo/python-list
Re: Down with tinyurl! (was Re: importing excel data into a python matrix?)
On Sep 19, 2010, at 6:05 PM, Xavier Ho wrote: On 20 September 2010 07:59, Ken Watford kwatford+pyt...@gmail.comkwatford%2bpyt...@gmail.com wrote: Not that I disagree with you, but you might find this helpful: http://tinyurl.com/preview.php -- http://mail.python.org/mailman/listinfo/python-list I don't think the OP wants a preview feature. The fact that you still have to go through tinyurl (which by the way, the short link itself in the email has no information on where it is whatsoever), and it makes searching through the archives difficult, too. We don't have a 140 character limit like Twitter. There's no reason we can't post the full link for reference purposes. Some email systems still insert hard line breaks around the 72 or 80 column mark and as a result long URLs get broken. I hope anyone on this list would be able to surgically repair a broken URL, but I email plenty of people who can't and tinyurl friends are really helpful in that context. bye P -- http://mail.python.org/mailman/listinfo/python-list
Re: Standard library function to remove folder with files and subfolders?
On Sep 18, 2010, at 5:24 PM, pyt...@bdurham.com wrote: Is there a standard library function to remove a folder that may contain files and subfolders? Or must I write a function that walks my folder's children and deletes all files/subfolders first, and then call os.removedirs()? In Python 2.x: shutil.rmtree() Might have moved in Python 3.x. HTH Philip -- http://mail.python.org/mailman/listinfo/python-list
