Re: [Python-Dev] epoll implementation

2006-05-27 Thread Martin v. Löwis
Ross Cohen wrote:
 epoll also allows 64 bits of data to be tucked away and returned when events
 happen. Could be useful for saving a dict lookup for every event. However,
 there are some refcounting issues. Dict lookup per event could be traded
 for one on deregistration. All it needs is a small forward-compatible
 extension to the current select.poll API.

I don't understand. You only need the dictionary on registration, to
find out whether this is a new registration or a modification of an
existing one. How could you save the dict lookup at registration time?
And what would you store in the additional data?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] warnings about missing __init__.py in toplevel directories

2006-05-27 Thread Martin v. Löwis
Ronald Oussoren wrote:
 Some time ago a warning was introduced for directories on sys.path  
 that don't contain an __init__.py but have the same name as a package/ 
 module that is being imported.
 
 Is it intentional that this triggers for toplevel imports? These  
 warnings are triggered in the build process for PyObjC, which is in  
 itself pretty harmless but very annoying.

They were very close to not being harmless earlier this year: Python
was almost changed to actually treat the directory as a package
even if it does not contain an __init__.py. In that case, the directory
itself would have been imported, not the thing inside. The warning
is (also) a hint that you should do some renaming - future versions
of Python might drop the need for __init__.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] epoll implementation

2006-05-27 Thread Martin v. Löwis
Greg Ewing wrote:
 There are many different select-like things around now
 (select, poll, epoll, kqueue -- are there others?) and
 random combinations of them seem to be available on any
 given platform. This makes writing platform-independent
 code that needs select-like functionality very awkward.
 
 Rather than adding yet another platform-dependent module,
 I'd like to see a unified Python interface in the stdlib
 that uses whichever is the best one available.

For epoll, that interface seems to be select.poll.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] epoll implementation

2006-05-27 Thread Martin v. Löwis
Alex Martelli wrote:
 Of course that would mean establishing which *was* the best available
 which, as we've seen this week, may not be easy.
 
 I believe it's: kqueue on FreeBSD ...

Such a statement assumes they are semantically equivalent. However, I
believe they aren't. A specific usage pattern can be expressed with
any of them, for such a usage pattern, you can come up with a ranking
even.

For example, to check whether a single specific file descriptor is
ready, I would assume that poll is best (even better than epoll and
kqueue). To check whether file descriptor 0 is ready, I would assume
that select is even better.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] epoll implementation

2006-05-27 Thread Ross Cohen
On Sat, May 27, 2006 at 08:36:12AM +0200, Martin v. Löwis wrote:
 Ross Cohen wrote:
  epoll also allows 64 bits of data to be tucked away and returned when events
  happen. Could be useful for saving a dict lookup for every event. However,
  there are some refcounting issues. Dict lookup per event could be traded
  for one on deregistration. All it needs is a small forward-compatible
  extension to the current select.poll API.
 
 I don't understand. You only need the dictionary on registration, to
 find out whether this is a new registration or a modification of an
 existing one. How could you save the dict lookup at registration time?
 And what would you store in the additional data?

The first thing any user of the poll interface does with the file descriptor
is map it to some state object. That's where the lookup can be saved, the
object can just be handed back directly. Problem being that when the fd is
unregistered, we won't get back the PyObject pointer so can't decrement
the refcount, has to be stored and looked up manually.

Ross
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] epoll implementation

2006-05-27 Thread Ross Cohen
On Sat, May 27, 2006 at 02:27:20AM +0100, Steve Holden wrote:
 Greg Ewing wrote:
  Rather than adding yet another platform-dependent module,
  I'd like to see a unified Python interface in the stdlib
  that uses whichever is the best one available.
  
 Of course that would mean establishing which *was* the best available 
 which, as we've seen this week, may not be easy.

Lots of systems (Windows? what's that?) implement both select and poll. I
have yet to see a system which implements more than 1 of epoll, kqueue,
/dev/poll, I/O completion ports, etc.

You'd have to search long and hard for a case where the non-select/poll
interfaces didn't perform better. Correctness/completeness is, of course,
a stronger criterion than performance, so if kqueue on OS X doesn't support
all relevant types of file descriptors, then it shouldn't used.

Fortunately, the python select.poll interface is a good match for all of
epoll, kqueue and /dev/poll. kqueue supports watching a lot more than file
descriptors, but that doesn't matter much for this discussion. kqueue also
mashes register/unregister/poll into a single call, which contributes to
how confusing the interface is, though from what I can tell it really is
basically the same as epoll.

As for Windows, the WSASelect interface is truly awful. The python library
currently sets it so you can pass in up to 512 descriptors, but don't count
on getting even that many. There is a hard limit on the number of sockets
a *user* can have open (varies based on OS version) with WSASelect and it
will just start returning WSAENOBUFS once you go over. You then have the
task of guessing how many you need to close before it will start working
again. Firewalls especially can eat into the pool.

IOCP has neither the WSAENOBUFS problem nor the 512 descriptor limit. It
would be nice (if you or your customers run Windows) if someone did a
select.poll interface with IOCP as a backend.

Ross
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] epoll implementation

2006-05-27 Thread Greg Ewing
Alex Martelli wrote:
 On May 26, 2006, at 6:27 PM, Steve Holden wrote:

Of course that would mean establishing which *was* the best available
which, as we've seen this week, may not be easy.
 
 I believe it's: kqueue on FreeBSD (for recent-enough versions  
 thereof), otherwise epoll where available and nonbuggy, otherwise  
 poll ditto, otherwise select

It would be an improvement if it would just pick *some*
implementation that worked, even if it weren't strictly
the best.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] epoll implementation

2006-05-27 Thread Martin v. Löwis
Ross Cohen wrote:
 The first thing any user of the poll interface does with the file descriptor
 is map it to some state object. That's where the lookup can be saved, the
 object can just be handed back directly. Problem being that when the fd is
 unregistered, we won't get back the PyObject pointer so can't decrement
 the refcount, has to be stored and looked up manually.

Ah. That would be an interface change; I would not do that. The lookup
is usually-constant and very fast. It is complexity that grows with the
number of file descriptors that these APIs worry about.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Need for Speed Sprint status

2006-05-27 Thread Tim Peters
 http://wiki.python.org/moin/NeedForSpeed/Successes
 http://wiki.python.org/moin/NeedForSpeed/Failures
 http://wiki.python.org/moin/NeedForSpeed/Deferred

And

http://wiki.python.org/moin/ListOfPerformanceRelatedPatches

All of these are linked to from the top page:

http://wiki.python.org/moin/NeedForSpeed

We still have 8 hours of sprinting today.  I'm sure we'll run
before+after benchmarks (at least pystone + pybench) on a variety of
boxes, and record the results, before it's over.  Many speedups won't
be reflected in that (e.g., major struct-module speedups 
enhancements, various string-integer speedups, major psyco speedups,
...).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] [Python-checkins] This

2006-05-27 Thread Martin v. Löwis
Terry Reedy wrote:
 Just end user experience's two cents here
 (btw, this line is correct  at English level?)
 
 Since you asked...your question would be better written is this line 
 correct English?
 And the line before, while not formal English of the kind needed, say, for 
 Decimal docs, works well enough for me as an expression of an informal 
 conversational thought.

Wouldn't it be still be conventional to have an article somewhere?
e.g.  Just /some/ end user's two cents here (also, isn't two cents
and experience roughly the same thing, so that one is redundant?)

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Need for Speed Sprint status

2006-05-27 Thread Steve Holden
Neal Norwitz wrote:
 First off, good work to everyone involved.  You did a tremendous job.
 I just hope to hell you're done, because I can't keep up! :-)
 
Not quite done yet, but I will be encouraging the team to start wrapping 
up in time to draw a line under everything that *isn't* going to 
continue by 17:00 GMT today, when we all turn into pumpkins.

 It would help me enormously if someone could summarize the status and
 everything that went on.  These are the things that would help me the
 most.
 
  * What are the speed diffs before/after the sprint
  * What was modified (summary)
  * What is left to do
 - doc
 - tests
 - code
  * Which branches are still planning to remain active
  * Lessons learned, how we can improve for the next time
  * Suggestions for further areas to look into improving
 
One of today's most important missions for me is to make sure that the 
results of the sprint are adequately documented, including pointers to 
further work that might yield speedups.

 It looks like there were a lot of additions to the string test suite,
 that's great.  I'm not sure if the other areas touched got similar
 boosts to their tests.  It would be good to upgrade all tests to
 verify corner cases of the implementation.  These tests should also be
 documented that they are to test the implementation rather than the
 language spec.  We don't want to write obscure tests that can't pass
 in other impls like Jython, IronPython, or PyPy.
 
Yes, we could really do with implementation-specific tests for this stuff.

 I will turn my amd64 box back on tomorrow and will also run Python
 through valgrind and pychecker when I get a chance.  There are a
 couple of new Coverity complaints that need to be addressed.
 
That'll be great. Again, let me say we've had terrific support from many 
other developers. I hope the whole community benefits from this sprint.

regards
  Steve
-- 
Steve Holden   +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd  http://www.holdenweb.com
Love me, love my blog  http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Proposal for a new itertools function: iwindow

2006-05-27 Thread Aahz
On Thu, May 25, 2006, Torsten Marek wrote:

 Some open question remain:
 - should iwindow return lists or tuples?
 - what happens if the length of the iterable is smaller than the window size,
 and no padding is specified? Is this an error? Should the generator return no
 value at all or one window that is too small?

You should probably try this idea out on comp.lang.python; if there is
approval, you'll probably need to write a PEP because of these issues.
Note that my guess is that the complexity of the issues relative to the
benefit means that BDFL will probably veto it.

Side note: if you do want to discuss this further on python-dev, please
subscribe
-- 
Aahz ([EMAIL PROTECTED])   * http://www.pythoncraft.com/

I saw `cout' being shifted Hello world times to the left and stopped
right there.  --Steve Gonedes
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] A Horrible Inconsistency

2006-05-27 Thread Aahz
On Fri, May 26, 2006, Fredrik Lundh wrote:

 and while we're at it, let's fix this:
 
   0.66 * (1, 2, 3)
  (1, 2)
 
 and maybe even this
 
   0.5 * (1, 2, 3)
  (1, 1)
 
 but I guess the latter one might need a pronunciation.

This should certainly get fixed in 3.0 thanks to __index__
-- 
Aahz ([EMAIL PROTECTED])   * http://www.pythoncraft.com/

I saw `cout' being shifted Hello world times to the left and stopped
right there.  --Steve Gonedes
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] [Python-checkins] r46300 - in python/trunk: Lib/socket.py Lib/test/test_socket.py Lib/test/test_struct.py Modules/_struct.c Modules/arraymodule.c Modules/socketmodule.c

2006-05-27 Thread Bob Ippolito
On May 26, 2006, at 4:56 PM, Guido van Rossum wrote:

 On 5/26/06, martin.blais [EMAIL PROTECTED] wrote:
 Log:
 Support for buffer protocol for socket and struct.

 * Added socket.recv_buf() and socket.recvfrom_buf() methods, that  
 use the buffer
   protocol (send and sendto already did).

 * Added struct.pack_to(), that is the corresponding buffer  
 compatible method to
   unpack_from().

 Hm... The file object has a similar method readinto(). Perhaps the
 methods introduced here could follow that lead instead of using two
 different new naming conventions?

(speaking specifically about struct and not socket)

pack_to and unpack_from are named as such because they work with objects
that support the buffer API (not file-like-objects). I couldn't find any
existing convention for objects that manipulate buffers in such a way.
If there is an existing convention then I'd be happy to rename these.

readinto seems to imply that some kind of position is being  
incremented.
Grammatically it only works if it's implemented on all buffer  
objects, but
in this case it's implemented on the Struct type.

-bob

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] warnings about missing __init__.py in toplevel directories

2006-05-27 Thread Ronald Oussoren

On 27-mei-2006, at 8:49, Martin v. Löwis wrote:

 Ronald Oussoren wrote:
 Some time ago a warning was introduced for directories on sys.path
 that don't contain an __init__.py but have the same name as a  
 package/
 module that is being imported.

 Is it intentional that this triggers for toplevel imports? These
 warnings are triggered in the build process for PyObjC, which is in
 itself pretty harmless but very annoying.

 They were very close to not being harmless earlier this year: Python
 was almost changed to actually treat the directory as a package
 even if it does not contain an __init__.py. In that case, the  
 directory
 itself would have been imported, not the thing inside. The warning
 is (also) a hint that you should do some renaming - future versions
 of Python might drop the need for __init__.

I'd do the renaming even if the warning would stay as a hint for  
users that forgot to create the __init__ file, I'm trying to keep my  
code warning free. I asked just to make sure that the warning is  
intentional for toplevel packages (which it is), I vaguely recall  
that there was some discussion about enabling the warning only for  
subpackages.

Ronald

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] getting rid of confusing expected a character buffer object messages

2006-05-27 Thread Fredrik Lundh
several string methods accepts either strings or objects that support 
the buffer api, and ends up raising a expected a character buffer 
object error if you pass in something else.  this isn't exactly helpful 
for non-experts -- the term character buffer object doesn't appear in 
any python tutorials I've seen.

  x.find(1)
 Traceback (most recent call last):
   File stdin, line 1, in module
 TypeError: expected a character buffer object


I think should be fixed, but I cannot make up my mind on what's the best 
way to fix it:

 1. map all errors from PyObject_AsCharBuffer to expected string
or compatible object, or some such.

 2. modify PyObject_AsCharBuffer so it returns different negative
error codes depending on whether the object doesn't support the
protocol at all, or if something went wrong when using it.

 3. to avoid changing the PyObject_AsCharBuffer return value, add a
PyObject_AsCharBufferEx that does the above

 4. do something entirely different

 5. fugetabotit

comments?

/F

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Proposal for a new itertools function: iwindow

2006-05-27 Thread Giovanni Bajo
Aahz [EMAIL PROTECTED] wrote:

 Some open question remain:
 - should iwindow return lists or tuples?
 - what happens if the length of the iterable is smaller than the
 window size, and no padding is specified? Is this an error? Should
 the generator return no value at all or one window that is too small?

 You should probably try this idea out on comp.lang.python; if there is
 approval, you'll probably need to write a PEP because of these issues.
 Note that my guess is that the complexity of the issues relative to
 the benefit means that BDFL will probably veto it.


A PEP for adding a simple generator to a library of generators??

The function does look useful to me. It's useful even if it returns a tuple
instead of a list, or if it doesn't support padding. It's still better than
nothing, and can still be succesfully used even if it requires some smallish
wrapper to achieve the exact functionality. I know I have been implementing
something similar very often. Since when do we need a full PEP process,
nitpicking the small details to death, just to add a simple function?

Giovanni Bajo

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Proposal for a new itertools function: iwindow

2006-05-27 Thread Raymond Hettinger
 Aahz [EMAIL PROTECTED] wrote:
 Some open question remain:
 - should iwindow return lists or tuples?
 - what happens if the length of the iterable is smaller than the
 window size, and no padding is specified? Is this an error? Should
 the generator return no value at all or one window that is too small?


An itertools windowing function was previously discussed and rejected.


Raymond
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] getting rid of confusing expected a character bufferobject messages

2006-05-27 Thread Terry Reedy

Fredrik Lundh [EMAIL PROTECTED] wrote in message 
news:[EMAIL PROTECTED]
 several string methods accepts either strings or objects that support
 the buffer api, and ends up raising a expected a character buffer
 object error if you pass in something else.  this isn't exactly helpful
 for non-experts -- the term character buffer object doesn't appear in
 any python tutorials I've seen.

  x.find(1)
 Traceback (most recent call last):
   File stdin, line 1, in module
 TypeError: expected a character buffer object


 I think should be fixed, but I cannot make up my mind on what's the best

'expected a string or character buffer' would be an improvement.

tjr



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] [Python-checkins] This

2006-05-27 Thread Terry Reedy

Martin v. Löwis [EMAIL PROTECTED] wrote in message 
news:[EMAIL PROTECTED]
 Terry Reedy wrote:
 Just end user experience's two cents here
 (btw, this line is correct  at English level?)

 Since you asked...your question would be better written is this line
 correct English?
 And the line before, while not formal English of the kind needed, say, 
 for
 Decimal docs, works well enough for me as an expression of an informal
 conversational thought.

 Wouldn't it be still be conventional to have an article somewhere?

Yes, as well as a verb...

 e.g.  Just /some/ end user's two cents here (also, isn't two cents
 and experience roughly the same thing, so that one is redundant?)

Maybe However, Facundo's compressed idiom struck me as an 
enter-the-conversation gambit rather than a convey-some-facts statements. 
So trying to pick it apart a piece at a time, with logic, does not work 
very well.Hence I restricted my intended-to-be-helpful suggestions to 
simple changes to the question itself and and the following statement.

Terry Jan Reedy




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Need for Speed Sprint status

2006-05-27 Thread Terry Reedy

Steve Holden [EMAIL PROTECTED] wrote in message 
news:[EMAIL PROTECTED]
 It looks like there were a lot of additions to the string test suite,
 that's great.  I'm not sure if the other areas touched got similar
 boosts to their tests.  It would be good to upgrade all tests to
 verify corner cases of the implementation.  These tests should also be
 documented that they are to test the implementation rather than the
 language spec.  We don't want to write obscure tests that can't pass
 in other impls like Jython, IronPython, or PyPy.

 Yes, we could really do with implementation-specific tests for this 
 stuff.

Just a reminder, in mid-April, there was a short thread on Python-3000 (to 
be continued here) entitled Separating out CPython and core Python tests,

I suggested separate files in separate directories.  Brett C. suggested, I 
believe, decorators to mark tests within a file.  Guido suggested maybe 
both, as appropriate.  The plan was to defer marking or movement of 
existing test until 2.6, but lets clearly comment any new 
implementation-specific tests as such.

Terry Jan Reedy





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] [Python-checkins] This

2006-05-27 Thread Greg Ewing
Martin v. Löwis wrote:
 
Just end user experience's two cents here
(btw, this line is correct  at English level?)

 Wouldn't it be still be conventional to have an article somewhere?
 e.g.  Just /some/ end user's two cents here

Or Just two cents' worth of end-user experience here,
which is almost as concise as the original and much
easier to parse.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Proposal for a new itertools function: iwindow

2006-05-27 Thread Nick Coghlan
Raymond Hettinger wrote:
 Aahz [EMAIL PROTECTED] wrote:
 Some open question remain:
 - should iwindow return lists or tuples?
 - what happens if the length of the iterable is smaller than the
 window size, and no padding is specified? Is this an error? Should
 the generator return no value at all or one window that is too small?
 
 
 An itertools windowing function was previously discussed and rejected.

A python-dev Google search for itertools window found me your original 
suggestion to include Jack Diedrich's itertools.window in Python 2.3 (which 
was only deferred because 2.3 was already past beta 1 at that point).

I couldn't find any discussion of the idea after that (aside from your 
pointing out that you'd never found a real-life use case for the pairwise() 
recipe in the docs, which is a basic form of windowing).

One option would be to add a windowing function to the recipes in the 
itertools docs. Something like:


def window(iterable, window_len=2, window_step=1):
 iterators = tee(iterable, window_len)
 for skip_steps, itr in enumerate(iterators):
 for ignored in islice(itr, skip_steps):
 pass
 window_itr = izip(*iterators)
 if window_step != 1:
 window_itr = islice(window_itr, step=window_step)
 return window_itr

As you can see, I'd leave the padding out of this function signature - if you 
want padding you can more easily specify it directly when calculating the 
iterable to be windowed. Do you want padding before and after the sequence, or 
just after? Pad with None or with zeroes? Better to expand on the padnone 
recipe:

def pad(iterable, pad_value=None, times=None):
 Pads an iterable with a repeating value

The quantity of padding may be limited by specifying
a specific number of occurrences.
 
 return chain(iterable, repeat(pad_value, times))

def padwindow(iterable, window_len, pad_value=None):
 Pads an iterable appropriately for the given window length

Padding is added to both the front and back of the sequence
such that the first and last windows will each contain one
value from the actual sequence.
 
 return chain(repeat(pad_value, window_len-1),
  iterable,
  repeat(pad_value, window_len-1))


Cheers,
Nick.

-- 
Nick Coghlan   |   [EMAIL PROTECTED]   |   Brisbane, Australia
---
 http://www.boredomandlaziness.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Proposal for a new itertools function: iwindow

2006-05-27 Thread Raymond Hettinger
From: Nick Coghlan [EMAIL PROTECTED]
 A python-dev Google search for itertools window found me your original 
 suggestion to include Jack Diedrich's itertools.window in Python 2.3 (which 
 was only deferred because 2.3 was already past beta 1 at that point).

 I couldn't find any discussion of the idea after that (aside from your 
 pointing out that you'd never found a real-life use case for the pairwise() 
 recipe in the docs, which is a basic form of windowing).

 One option would be to add a windowing function to the recipes in the 
 itertools docs. Something like:


 def window(iterable, window_len=2, window_step=1):
 iterators = tee(iterable, window_len)
 for skip_steps, itr in enumerate(iterators):
 for ignored in islice(itr, skip_steps):
 pass
 window_itr = izip(*iterators)
 if window_step != 1:
 window_itr = islice(window_itr, step=window_step)
 return window_itr

No thanks.  The resolution of this one was that windowing iterables is not a 
good idea.  It is the natural province of sequences, not iterables.  With 
sequences, it is a matter of using an index and offset.  With iterables, there 
is a great deal of data shifting.  Also note that some of the uses are subsumed 
by collections.deque().  In implementing a draft of itertools window, we found 
that the data shifting was unnatural and unavoidable (unless you output some 
sort of buffer instead of a real tuple).  Also, we looked at use cases and 
found 
that most had solutions that were dominated by some other approach.  The 
addition of a windowing tool would tend to steer people away from better 
solutions.  In short, after much deliberation and experimenting with a sample 
implementation, the idea was rejected.  Hopefully, it will stay dead and no one 
will start a cruscade for it simply because it can be done and because it seems 
cute.

The thought process was documented in a series of newsgroup postings:
http://groups.google.com/group/comp.lang.python/msg/026da8f9eec4becf

The SF history is less informative because most of the discussions were held by 
private email:
   http://www.python.org/sf/756253

If someone aspires to code some new itertools, I have approved two new ones, 
imerge() and izip_longest().  The pure python code for imerge() is at:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/491285

The impetus behind izip_longest() was expressed in a newsgroup thread, 
http://groups.google.com/group/comp.lang.python/browse_thread/thread/f424f63bfdb77c4/38c31991133757f7
 . 
The discussion elicted neither broad support, nor condemnation.  Also, the use 
cases were sparse.  Ultimately, I was convinced by encountering a couple of 
natural use cases.  Another thought is that while other solutions are usually 
available for any given use case, there is a natural tendency to reach for a 
zip-type tool whenever presented with lock-step iteration issues, even when the 
inputs are of uneven length.  Note, that the signature for izip_longest() needs 
to include an optional pad value (defaulting to None) -- there are plenty of 
use 
cases where an empty string, zero, or a null object would be a preferred pad 
value.

Cheers,



Raymond 

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com