Re: Is Python Lazy?
Dan Stromberg, 06.05.2012 07:40: > you probably won't be able to write a sort routine, and > use it as a "first 100 lowest values" routine for free. But you could > construct something that does almost the same thing lazily using a > generator - not for free. OTOH, if you really wanted to do this particular thing, I'd just use heapq. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Is Python Lazy?
Generators and iterators are laziness where you tend to need laziness the most. Generator expressions are tiny generators - more full fledged generators are supported. Python probably won't have laziness at its core ever, but it's nice having a dose of it. IOW, you probably won't be able to write a sort routine, and use it as a "first 100 lowest values" routine for free. But you could construct something that does almost the same thing lazily using a generator - not for free. On Sat, May 5, 2012 at 9:19 PM, Emeka wrote: > > Hello All, > > Could one say that generator expressions and functions are Python way of > introducing Lazy concept? > > Regards, \Emeka > -- > *Satajanus Nig. Ltd > > > * > > -- > http://mail.python.org/mailman/listinfo/python-list > > -- http://mail.python.org/mailman/listinfo/python-list
Is Python Lazy?
Hello All, Could one say that generator expressions and functions are Python way of introducing Lazy concept? Regards, \Emeka -- *Satajanus Nig. Ltd * -- http://mail.python.org/mailman/listinfo/python-list
Re: pyjamas / pyjs
On May 4, 11:43 pm, Duncan Booth wrote: > In case it isn't obvious why I might be subscribed but emails turned off, I > read mailing lists like that through gmane in which case I still need to > sign up to the list to post but definitely don't want to receive emails. This. I was surprised to suddenly start receiving emails, as I thought I'd left the pyjamas list _years_ ago. I've asked them to stop spamming me and even now and getting snide, shitty emails claiming that what they were doing wasn't spam as I "could have chosen to ignore it or delete it". I've had to email the abuse address at Rackspace. I'm pretty sure it's illegal to add people to a mailing list without their consent, to not include instructions on how to unsubscribe from the mailing list, and to continue to email them when they've asked to be removed. As for the project itself, having a bunch of arrogant assholes decide that the lead developer has "hijacked" the project with his explicit- from-the-start libre software "philosophies", because such belief runs counter to the desires - sorry, the "philosophies" - held by that core group of assholes, is just the must astounding hypocrisy I've ever seen. The correct approach in such an instance is to *fork* the project; but that way you don't get to steal the community, I guess. The unwanted email was bad enough. The overwhelming sense of entitlement those emails expressed was even worse. -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with time.time() standing still
On Sun, May 6, 2012 at 6:51 AM, Bob Cowdery wrote: > The time.clock() function does increment correctly. CPU is around 30% 30% of how many cores? If that's a quad-core processor, that could indicate one core completely pegged plus a little usage elsewhere. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with time.time() standing still
On 05May2012 20:33, Bob Cowdery wrote: | I've written a straight forward extension that wraps a vendors SDK for a | video capture card. All works well except that in the Python thread on | which I call the extension, after certain calls that I believe are using | DirectShow, time stands still. The thread returns fine as the extension | does its stuff in its own threads. In all other respects the Python | thread seems unaffected but calls to time.time() always return the same | time which is usually several seconds in the past or future and always | has no fractional part. Thought #1: you are calling time.time() and haven't unfortunately renamed it? (I doubt this scenario, though the lack of fractional part is interesting.) | If I leave it long enough time will suddently | jump forward after a few minutes, then again after a few minutes more. | | I've never encountered this behaviour before and can't understand what | on earth is going on. If I call the 'C' time() function just the other | side of my call to the extension the time is ticking along fine. It's | just the one Python thread that is affected. [...] Thought #2: On a UNIX system I'd be running the process under strace (or dtrace or ktrace depending on flavour) to see what actual OS system calls are being made during this behaviour. Is this feasible for you? Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ A slipping sear could let your M203 grenade launcher fire when you least expect it. That would make you quite unpopular in what's left of your unit. - page 9 of the August 1993 issue of PS magazine, the US Army's preventive maintenance magazine -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with time.time() standing still
Thanks Daniel, that's interesting. Unfortunately there is no sensible code I can post because this only happens when I make a specific call into the vendors SDK. I can exercise my own code in the extension without a problem. The python test calling code is doing practically nothing. I make 3 calls to the extension to set things going and then loop every 5 seconds and print the time. I know Pythons own threading model is cooperative because of the GIL and therefore one thread can hog the show but I've never had issues with threads running in a C extension messing up a Python thread. I really need to understand what mechanism is at play here rather than work around it. Bob The time.clock() function does increment correctly. CPU is around 30% On 05/05/2012 21:17, Danyel Lawson wrote: > Add a time.sleep(0) call to all your loops. Multithreading in Python > is a cooperative cross platform threading simulation if you have tight > loops Python won't task switch until you make a system call. > Potentially preventing internal library variables from being updated. > > Your five minute interval may be almost exactly how long it takes to > process a flooded queue in a tight loop in your program and it may be > why it continues to happen as the queue waits to fill again while > processing happens. > > You can simulate the progression of time by overriding the time.time > function by simply setting it to a function that just increments a > module level or function property variable. You can also override the > time.time function to return the posix time function's value to maybe > get around whatever optimization is happening in the time.time > function to pass back the same value. > > If you post your sample code that exhibits the same behavior it may be > obvious to someone on the list as to what is the problem. > > > > On Sat, May 5, 2012 at 3:33 PM, Bob Cowdery wrote: >> Hi all, >> >> I've been a long time user of Python and written many extensions but >> this problem has me stumped. >> >> I've written a straight forward extension that wraps a vendors SDK for a >> video capture card. All works well except that in the Python thread on >> which I call the extension, after certain calls that I believe are using >> DirectShow, time stands still. The thread returns fine as the extension >> does its stuff in its own threads. In all other respects the Python >> thread seems unaffected but calls to time.time() always return the same >> time which is usually several seconds in the past or future and always >> has no fractional part. If I leave it long enough time will suddently >> jump forward after a few minutes, then again after a few minutes more. >> >> I've never encountered this behaviour before and can't understand what >> on earth is going on. If I call the 'C' time() function just the other >> side of my call to the extension the time is ticking along fine. It's >> just the one Python thread that is affected. If I call from the main >> thread then the main thread is affected. The calling program I've used >> to test this is just a few lines long. >> >> I believe the time function is a thin wrapper over the 'C' runtime so >> its very odd that time only stands still on the Python side but not on >> the 'C' side. As time is built in I've not looked at the code as its not >> in the distribution. Don't know if it would help to do that. >> >> This also can affect the time.sleep() function making it return >> immediately but that only seems to happen in my full application. >> >> Any ideas would be very greatly received. >> >> Bob >> -- >> http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with time.time() standing still
Add a time.sleep(0) call to all your loops. Multithreading in Python is a cooperative cross platform threading simulation if you have tight loops Python won't task switch until you make a system call. Potentially preventing internal library variables from being updated. Your five minute interval may be almost exactly how long it takes to process a flooded queue in a tight loop in your program and it may be why it continues to happen as the queue waits to fill again while processing happens. You can simulate the progression of time by overriding the time.time function by simply setting it to a function that just increments a module level or function property variable. You can also override the time.time function to return the posix time function's value to maybe get around whatever optimization is happening in the time.time function to pass back the same value. If you post your sample code that exhibits the same behavior it may be obvious to someone on the list as to what is the problem. On Sat, May 5, 2012 at 3:33 PM, Bob Cowdery wrote: > Hi all, > > I've been a long time user of Python and written many extensions but > this problem has me stumped. > > I've written a straight forward extension that wraps a vendors SDK for a > video capture card. All works well except that in the Python thread on > which I call the extension, after certain calls that I believe are using > DirectShow, time stands still. The thread returns fine as the extension > does its stuff in its own threads. In all other respects the Python > thread seems unaffected but calls to time.time() always return the same > time which is usually several seconds in the past or future and always > has no fractional part. If I leave it long enough time will suddently > jump forward after a few minutes, then again after a few minutes more. > > I've never encountered this behaviour before and can't understand what > on earth is going on. If I call the 'C' time() function just the other > side of my call to the extension the time is ticking along fine. It's > just the one Python thread that is affected. If I call from the main > thread then the main thread is affected. The calling program I've used > to test this is just a few lines long. > > I believe the time function is a thin wrapper over the 'C' runtime so > its very odd that time only stands still on the Python side but not on > the 'C' side. As time is built in I've not looked at the code as its not > in the distribution. Don't know if it would help to do that. > > This also can affect the time.sleep() function making it return > immediately but that only seems to happen in my full application. > > Any ideas would be very greatly received. > > Bob > -- > http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Problem with time.time() standing still
Hi all, I've been a long time user of Python and written many extensions but this problem has me stumped. I've written a straight forward extension that wraps a vendors SDK for a video capture card. All works well except that in the Python thread on which I call the extension, after certain calls that I believe are using DirectShow, time stands still. The thread returns fine as the extension does its stuff in its own threads. In all other respects the Python thread seems unaffected but calls to time.time() always return the same time which is usually several seconds in the past or future and always has no fractional part. If I leave it long enough time will suddently jump forward after a few minutes, then again after a few minutes more. I've never encountered this behaviour before and can't understand what on earth is going on. If I call the 'C' time() function just the other side of my call to the extension the time is ticking along fine. It's just the one Python thread that is affected. If I call from the main thread then the main thread is affected. The calling program I've used to test this is just a few lines long. I believe the time function is a thin wrapper over the 'C' runtime so its very odd that time only stands still on the Python side but not on the 'C' side. As time is built in I've not looked at the code as its not in the distribution. Don't know if it would help to do that. This also can affect the time.sleep() function making it return immediately but that only seems to happen in my full application. Any ideas would be very greatly received. Bob -- http://mail.python.org/mailman/listinfo/python-list
Re: When convert two sets with the same elements to lists, are the lists always going to be the same?
> Documentation that takes ten pages to say something is just as bad as > documentation that leaves stuff out, because it's almost guaranteed > that it won't be read. That's the point. If a simple example (6 lines) can demonstrate the concept, why spending "ten pages" to explain it. My experience is that for certain things, it is better describe by a spec once you know it, but it is certainly not true for people to learn it. A reasonable strategy is to interleave spec with demonstrating examples. There is no excuse to not to make the manual easier to read. -- Regards, Peng -- http://mail.python.org/mailman/listinfo/python-list
Re: How to compute a delta: the difference between lists of strings
J. Mwebaze wrote: > This is out of curiosity, i know this can be done with python diffllib > module, but been figuring out how to compute the delta, Consider two lists > below. > > s1 = ['e', 'f', 'g', 'A', 'B', 'C', 'D', 'C'] > s2 =['e', 'A', 'B', 'f', 'g', 'C', 'D', 'z'] > > This is the result should be > > [' e', '+ A', '+ B', ' f', ' g', '- A', '- B', ' C', ' D', '- C', '+ > z'] > > ideas on how to approach this.. ? http://en.wikipedia.org/wiki/Longest_common_subsequence_problem -- ZeD -- http://mail.python.org/mailman/listinfo/python-list
Re: When convert two sets with the same elements to lists, are the lists always going to be the same?
On 5/5/2012 4:04 AM Peng Yu said... I agree that people have different opinions on issues like this. But I think that "The Customer Is God". Readers of the doc is the customers, the writers of the doc is the producers. The opinion of customers should carry more weight than producers. Only to a point. The experience of the producers should carry more weight than the opinion of the customers. Emile -- http://mail.python.org/mailman/listinfo/python-list
Re: How to compute a delta: the difference between lists of strings
On 5/5/2012 5:12 AM J. Mwebaze said...
This is out of curiosity, i know this can be done with python diffllib
module, but been figuring out how to compute the delta, Consider two
lists below.
s1 = ['e', 'f', 'g', 'A', 'B', 'C', 'D', 'C']
s2 =['e', 'A', 'B', 'f', 'g', 'C', 'D', 'z']
This is the result should be
[' e', '+ A', '+ B', ' f', ' g', '- A', '- B', ' C', ' D', '- C',
'+ z']
ideas on how to approach this.. ?
use difflib:
from difflib import unified_diff as ud
s1 = ['e', 'f', 'g', 'A', 'B', 'C', 'D', 'C']
s2 = ['e', 'A', 'B', 'f', 'g', 'C', 'D', 'z']
[ r for r in ud(s1,s2,lineterm="|") if not r.endswith("|") ]
HTH,
Emile
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to compute a delta: the difference between lists of strings
thank Chris.. On Sat, May 5, 2012 at 2:39 PM, Chris Angelico wrote:k > On Sat, May 5, 2012 at 10:12 PM, J. Mwebaze wrote: > > This is out of curiosity, i know this can be done with python diffllib > > module, but been figuring out how to compute the delta, Consider two > lists > > below. > > > > s1 = ['e', 'f', 'g', 'A', 'B', 'C', 'D', 'C'] > > s2 =['e', 'A', 'B', 'f', 'g', 'C', 'D', 'z'] > > > > This is the result should be > > > > [' e', '+ A', '+ B', ' f', ' g', '- A', '- B', ' C', ' D', '- C', '+ > > z'] > > > > ideas on how to approach this.. ? > > Here's a simple algorithm that produces sorta-mostly-reasonable > results most of the time: > > Set your current-position-index to the beginning of each lists. (Call > them 'pos1' and 'pos2'.) > If s1[pos1] and s2[pos2] are identical, match - increment and iterate. > Otherwise, increment pos2 until either it matches pos1 or you reach > the end of s2. > If you find a match, report the insertion into s2, increment both > pointers past the match, and carry on. > If you hit the end of s2, increment pos1 once and report an insertion > into s1, then go back to looking for a match. > > It helps to append a sentinel to each list; that way, you don't need > to separately check for additional content at the end of either list > (as the sentinel won't match any of the strings). > > This is the algorithm I used for writing a simple file diff tool ages > ago. It's not as good as diff(1), but it was fun to do the exercise. > It's quite inefficient at handling large insertions into s1, and needs > to be modified for most file handling (for instance, requiring a > 2-line or 3-line rematch after a difference, to avoid matching on > blank lines), but it's a basis. > > Producing beautiful or minimal diffs is a more complex task. :) > > ChrisA > -- > http://mail.python.org/mailman/listinfo/python-list > -- *Mob UG: +256 (0) 70 1735800 | NL +31 (0) 6 852 841 38 | Gtalk: jmwebaze | skype: mwebazej | URL: www.astro.rug.nl/~jmwebaze /* Life runs on code */* -- http://mail.python.org/mailman/listinfo/python-list
Re: How to compute a delta: the difference between lists of strings
On Sat, May 5, 2012 at 10:12 PM, J. Mwebaze wrote: > This is out of curiosity, i know this can be done with python diffllib > module, but been figuring out how to compute the delta, Consider two lists > below. > > s1 = ['e', 'f', 'g', 'A', 'B', 'C', 'D', 'C'] > s2 =['e', 'A', 'B', 'f', 'g', 'C', 'D', 'z'] > > This is the result should be > > [' e', '+ A', '+ B', ' f', ' g', '- A', '- B', ' C', ' D', '- C', '+ > z'] > > ideas on how to approach this.. ? Here's a simple algorithm that produces sorta-mostly-reasonable results most of the time: Set your current-position-index to the beginning of each lists. (Call them 'pos1' and 'pos2'.) If s1[pos1] and s2[pos2] are identical, match - increment and iterate. Otherwise, increment pos2 until either it matches pos1 or you reach the end of s2. If you find a match, report the insertion into s2, increment both pointers past the match, and carry on. If you hit the end of s2, increment pos1 once and report an insertion into s1, then go back to looking for a match. It helps to append a sentinel to each list; that way, you don't need to separately check for additional content at the end of either list (as the sentinel won't match any of the strings). This is the algorithm I used for writing a simple file diff tool ages ago. It's not as good as diff(1), but it was fun to do the exercise. It's quite inefficient at handling large insertions into s1, and needs to be modified for most file handling (for instance, requiring a 2-line or 3-line rematch after a difference, to avoid matching on blank lines), but it's a basis. Producing beautiful or minimal diffs is a more complex task. :) ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: When convert two sets with the same elements to lists, are the lists always going to be the same?
On Sat, May 5, 2012 at 9:04 PM, Peng Yu wrote: > I agree that people have different opinions on issues like this. But I > think that "The Customer Is God". Readers of the doc is the customers, > the writers of the doc is the producers. The opinion of customers > should carry more weight than producers. Nah, the customer's not God, and I have proof. http://notalwaysright.com/ Oops, now everyone's off reading funny stories about stupid/abusive customers. Well, when you're all back... The reason you're reading documentation is to learn. You're not handing over wads of cash and saying "Do stuff for me"; you're reading the Player's Handbook and learning which dice to roll when. Perhaps there's some jargon that you don't understand; in that case, either a FAQ/glossary or a forum post will set you straight. But if the doc says that something can't be relied upon, then there's nothing more to write there. Documentation that takes ten pages to say something is just as bad as documentation that leaves stuff out, because it's almost guaranteed that it won't be read. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: When convert two sets with the same elements to lists, are the lists always going to be the same?
Hi Terry, Thank you for you detailed email. > If two collections are equal, should the iteration order be the same? It has > always been true that if hash values collide, insertion order matters. > However, a good hash function avoids hash collisions as much as possible in > practical use cases. Without doing something artificial, as I did with the > example, collisions should be especially rare on 64-bit builds. If one > collection has a series of additions and deletions so that the underlying > hash table has a different size than an equal collection build just from > insertions, then order will also be different. The reason that I asked to add the artificial example in the doc is because I never completely understand the unorderness until I see your artificial example. You will surely use more words for explaining what "unorderness" means than just showing your example. And the example (since formatted as code) is more eye catching than just plain text. For my case, since I didn't understand the unorderness, I made some subtle bug in my program, which works fine in my testing code. However, it produce a small amount of corrupted results for the real production use, which is harder to debug. It did wasted quite some of my time. > For the doc, the problem is what to say and where without being repetitous > (and to get multiple people to agree ;-). I agree that people have different opinions on issues like this. But I think that "The Customer Is God". Readers of the doc is the customers, the writers of the doc is the producers. The opinion of customers should carry more weight than producers. -- Regards, Peng -- http://mail.python.org/mailman/listinfo/python-list
Re: key/value store optimized for disk storage
On Friday, 4 May 2012 16:27:54 UTC+1, Steve Howell wrote: > On May 3, 6:10 pm, Miki Tebeka wrote: > > > I'm looking for a fairly lightweight key/value store that works for > > > this type of problem: > > > > I'd start with a benchmark and try some of the things that are already in > > the standard library: > > - bsddb > > - sqlite3 (table of key, value, index key) > > - shelve (though I doubt this one) > > > > Thanks. I think I'm ruling out bsddb, since it's recently deprecated: > > http://www.gossamer-threads.com/lists/python/python/106494 > > I'll give sqlite3 a spin. Has anybody out there wrapped sqlite3 > behind a hash interface already? I know it's simple to do > conceptually, but there are some minor details to work out for large > amounts of data (like creating the index after all the inserts), so if > somebody's already tackled this, it would be useful to see their > code. > > > You might find that for a little effort you get enough out of one of these. > > > > Another module which is not in the standard library is hdf5/PyTables and in > > my experience very fast. > > Thanks. Could also look at Tokyo cabinet or Kyoto cabinet (but I believe that has slightly different licensing conditions for commercial use). -- http://mail.python.org/mailman/listinfo/python-list
