Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Danilo J. S. Bellini]

The proposal is mostly about scan/accumulate. Reduce/fold is a "corollary",
as it's just the last value of a scan. The idea is to have a way of using
the previous iteration output inside a list comprehension (and anything
alike). That is, to make them recursive.

last([abs(prev - x) for x in range(10) from prev = 2])
>
Why not [abs(prev - x) for x in range(10) from prev = 2][-1]?
How about list(some_iterable)[-1]?
Probably a "last" function would avoid these.

But the "last" is a pretty easy function to write. This proposal is about
the list comprehension syntax (and other things alike). The "last" function
and the "scan" functions can be seen as secondary proposals, the main point
is a syntax to access to the previous iteration output value inside a list
comprehension. For example, a product:

>>> [prev * k for k in [5, 2, 4, 3] from prev = 1]
[1, 5, 10, 40, 120]

That makes sense for me, and seem simpler than:

>>> from itertools import accumulate, chain
>>> list(accumulate(chain([1], [5, 2, 4, 3]), lambda prev, k: prev * k))
[1, 5, 10, 40, 120]

Which is still simpler than using reduce

>>> from functools import reduce
>>> list(reduce(lambda hist, k: hist + [hist[-1] * k], [5, 2, 4, 3], [1]))
[1, 5, 10, 40, 120]

The first is explicit. The imperative approach for that would be much more
like the reduce than the scan, as "hist" is the result.

>>> hist = [1]
>>> for k in [5, 2, 4, 3]:
... prev = hist[-1]
... hist.append(prev * k)
>>> hist
[1, 5, 10, 40, 120]

The very idea of prefering these approaches instead of the proposal sounds
strange to me. What is simpler on them, the McCabe complexity? Number of
tokens? Number of repeated tokens? AST tree height?

AFAIK, GvR prefers the list comprehension syntax instead of using the
map/filter higher order functions. He even said somewhere that a reduce can
be written as list comprehension, and it wasn't obvious for me that a
"3-for-sections" list comprehension repeating a target variable name would
be a valid Python code, and that's required to get a recursive list
comprehension. What I'm proposing is to allow a list comprehension syntax
to behave like itertools.accumulate without the "3-for-sections" kludge.

The rationale for the proposal is here:
https://github.com/danilobellini/pyscanprev/tree/v0.1.0#the-world-without-this-package-rationale

On Haskell, the above example would be:

Prelude> scanl (*) 1 [5, 2, 4, 3]
[1,5,10,40,120]

And that's what I'm trying to write as a list comprehension. Some months
ago, thinking on how I could write this proposal, I started writing
PyScanPrev. Among the examples I wrote on PyScanPrev, there are use cases
on:
- maths
- physics
- economics
- string processing
- signal processing
- control engineering
- dynamic / time-varying model simulation
- gray code generation

So I can say that's not niche/specific. The most sophisticated scan example
I wrote is this one to plot the trajectory of a "leaking
bucket-spring-damper" system:
https://github.com/danilobellini/pyscanprev/blob/v0.1.0/examples/state-space.rst

Lag and windowing seem unrelated, something that would be solved with
itertools.tee, zip or perhaps a collections.deque and a function (and I
remember of doing so on AudioLazy).


2016-10-23 22:38 GMT-02:00 Chris Angelico :

> On Mon, Oct 24, 2016 at 11:29 AM, Steven D'Aprano 
> wrote:
> > In fairness I am sick and if I were well I may have been able to keep
> > this straight in my head, but calling the variable "prev" is actively
> > misleading. I was mislead, and (I think) Chris who just suggested this
> > was similar to the SQL "lag" function may have been mislead as well. (Or
> > perhaps he was just mislead by me, in which case, sorry Chris!)
>
> All of the above are possible. I'm skimming the thread, not reading it
> in full, and I'm a bit lost as to the point of the proposal, so it's
> entirely possible that lag() is unrelated.
>
> But if it is indeed just reduce(), then it's even simpler.
>
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Danilo J. S. Bellini
---
"*It is not our business to set up prohibitions, but to arrive at
conventions.*" (R. Carnap)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Chris Angelico]

On Mon, Oct 24, 2016 at 11:29 AM, Steven D'Aprano  wrote:
> In fairness I am sick and if I were well I may have been able to keep
> this straight in my head, but calling the variable "prev" is actively
> misleading. I was mislead, and (I think) Chris who just suggested this
> was similar to the SQL "lag" function may have been mislead as well. (Or
> perhaps he was just mislead by me, in which case, sorry Chris!)

All of the above are possible. I'm skimming the thread, not reading it
in full, and I'm a bit lost as to the point of the proposal, so it's
entirely possible that lag() is unrelated.

But if it is indeed just reduce(), then it's even simpler.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Steven D'Aprano]

On Sun, Oct 23, 2016 at 02:10:42PM -0200, Danilo J. S. Bellini wrote:
> >
> > Ah, that makes SIX existing solutions. Do we need a seventh?
> 
> It might have dozens of solutions, perhaps an infinity of solutions.
> Brainfuck and assembly can be used, or even turing machine instructions...

No, those are *implementations*. You can implement your solution in any 
language you like. For integration with Python, any of C, Fortran, Rust, 
Julia, Cython and (of course) pure Python are proven to work well. Using 
Turing Machine instructions requires some sort of TM compiler... good luck with 
that.

But you cut out the most important part of my post. You've given lots of 
existing solutions. Why aren't they satisfactory? Even if somebody wants 
to write in a functional style, the reduce() solution you show seems 
perfectly clean and conventional to anyone used to functional code:

from functools import reduce
reduce(lambda prev, x: abs(prev - x), [3, 4, 5], 2)

returns 2. What is wrong with this solution? That is the obvious 
solution for somebody looking for a functional style: something called 
reduce or fold. And there's no harm in needing to import reduce. Not 
every function has to be a built-in.


Whereas your suggestion needs TWO new features: new syntax:

(abs(prev - x) for x in [3, 4, 5] from prev = 2)

plus a new built-in function last() which extracts the final value from 
an iterator. That means you will risk encouraging people to wastefully 
generate a large container of unneeded and unwanted intermediate values:

last([abs(prev - x) for x in range(10) from prev = 2])

which will generate a list 10 items long just to extract the final 
one. reduce() is better, that is exactly what reduce() is designed for.


> But there should be one, and preferably only one, OBVIOUS way to do it.
> Readability counts.

Right. And the obvious way is the imperative approach (only this time I 
will use a better variable name):

py> result = 2
py> for x in [3, 4, 5]:
... result = abs(result - x)
...
py> result
2


For those who think in functional programming terms, reduce() is the 
obvious way.



Also, I feel that your proposal could have been explained better. I felt 
overloaded by the sheer mass of different alternatives, and mislead by 
your use of the name "prev" for something that I see now on more careful 
reading is *not* the previous value of the loop variable (as I first 
understood) but the running calculation result.

In fairness I am sick and if I were well I may have been able to keep 
this straight in my head, but calling the variable "prev" is actively 
misleading. I was mislead, and (I think) Chris who just suggested this 
was similar to the SQL "lag" function may have been mislead as well. (Or 
perhaps he was just mislead by me, in which case, sorry Chris!)


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[David Mertz]

On Sun, Oct 23, 2016 at 4:22 PM, Steven D'Aprano 
wrote:

> Right. But you're missing the point of Danilo's proposal. He isn't
> asking for a function to "jump to the end" of an iterator. Look at his
> example. The word "last" is a misnomer: he seems to me talking
> about having a special variable in comprehensions that holds the
> *previous* value of the loop variable, with special syntax to set its
> FIRST value, before the loop is entered.


OK... but that's exactly itertools.accumulate (or maybe a thin wrapper
around it I like showed earlier in the thread).

I'm not sure Danilo was clear in what he's proposing.  In the thread he
suggested that he wanted to special case to indexing on sequences, which
doesn't seem to make sense for your meaning.

It feels like there might be a case here for a new function in itertools
that makes use of the last-seen item in an iterable, then combines it
somehow with the current item.  I'm not sure the spelling, but it
definitely sounds like a function to me, not a need for new syntax.  I've
only rarely had that specific need.

That said, here's a function I use in teaching to show some of what you can
do with combining iterators, especially using itertools:

def item_with_total(iterable):
"Generically transform a stream of numbers into a pair of (num,
running_sum)"
s, t = tee(iterable)
yield from zip(t, accumulate(s))


This might not be *exactly* what Danilo wants, but it's a similar concept.
I wrap together an iterator (including an infinite one) with an
accumulation.  This just takes the default `operator.add` function for
accumulate(), but it could take a function argument easily enough.

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Steven D'Aprano]

On Mon, Oct 24, 2016 at 10:22:32AM +1100, Steven D'Aprano wrote:
> On Sun, Oct 23, 2016 at 08:47:12AM -0700, David Mertz wrote:
> > Consuming the iterator is *necessary* to get the last item. There's no way
> > around that.
> > 
> > Obviously, you could itertools.tee() it first if you don't mind the cache
> > space. But there cannot be a generic "jump to the end" of an iterator
> > without being destructive.
> 
> Right. But you're missing the point of Danilo's proposal. 

Ah, actually it may be that I have misunderstood Danilo's proposal, 
because his example does include BOTH a suggestion of new magic syntax 
for retrieving the *previous* loop value inside a comprehension AND what 
seems to be a new built-in(?) function last() which seems to do exactly 
what you suggest: jump right to the end of an iterable and return the 
final value.

My apologies for the confusion. 


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Steven D'Aprano]

On Sun, Oct 23, 2016 at 08:47:12AM -0700, David Mertz wrote:
> Consuming the iterator is *necessary* to get the last item. There's no way
> around that.
> 
> Obviously, you could itertools.tee() it first if you don't mind the cache
> space. But there cannot be a generic "jump to the end" of an iterator
> without being destructive.

Right. But you're missing the point of Danilo's proposal. He isn't 
asking for a function to "jump to the end" of an iterator. Look at his 
example. The word "last" is a misnomer: he seems to me talking 
about having a special variable in comprehensions that holds the 
*previous* value of the loop variable, with special syntax to set its 
FIRST value, before the loop is entered. So "last" is a misleading name, 
unless you understand it as "last seen" rather than "very last, at the 
end".

So given an iterator [1, 2, 4, 8], and an initial value of -1, we would 
see something like this:

[(previous, this) for this in [1, 2, 4, 8] with previous as -1]
# or some other syntax

returns:

[(-1, 1), (1, 2), (2, 4), (4, 8)]

So a dedicated function that does nothing but scan to the end of the 
iterator and return the last/final value seen is no alternative to his 
proposal.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Paul Moore]

On 23 October 2016 at 17:10, Danilo J. S. Bellini
 wrote:
>> Ah, that makes SIX existing solutions. Do we need a seventh?
>
> It might have dozens of solutions, perhaps an infinity of solutions.
> Brainfuck and assembly can be used, or even turing machine instructions...
>
> But there should be one, and preferably only one, OBVIOUS way to do it.
> Readability counts.

Sure, but you haven't explained why your proposal is more obvious than
any of the other six. Built in does not equate to obvious. More
obvious is often to have a dedicated tool, in a module designed to
provide tools in that particular area. That's partially why reduce got
moved to the functools module (another part is the fact that Guido
doesn't find functional-style approaches that "obvious" - and what's
obvious to a Dutchman is the benchmark here :-))

I'm not against powerful "windowed function" capabilities - my
background is SQL, and windowed functions in SQL have even more power
than the sort of thing we're talking about here. But I wouldn't call
them "obvious" - at least not based on the number of times I get to do
explanations of them to colleagues, or the number of tutorials on them
I see. So the idea seems fine to me, but I'd very definitely class it
as an "advanced" feature, and typically that sort of feature in Python
is handled in a library.

> Reduce lost the built-in status on Python 3. Lambdas lost the decomposing
> arguments like "lambda (a, b), c: a + b * c".

Which can be interpreted as evidence that this type of approach is not
considered a core feature.

In general, I like the idea, but I don't think it fits well in Python
in its proposed form.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Danilo J. S. Bellini]

>
> I would have preferred this signature to start with, but it's easy to
> wrap.
>
Indeed, but a default value for the first argument requires a default value
for all arguments. It's a syntax error, but I agree a "range-like"
signature like that would be better.

My reference scan implementation (that's how I thought itertools.accumulate
should be):
https://github.com/danilobellini/pyscanprev/blob/v0.1.0/pyscanprev.py#L171

A new "functools.scan" with a signature like the one from the link above
would be nice, but it would overlap with itertools.accumulate in some
sense. The advantages would be:

1 - The scan signature and the functools.reduce signature are the same (the
function as the first parameter, like map/filter)
2 - The module, functools, is the same that has the reduce function

-- 
Danilo J. S. Bellini
---
"*It is not our business to set up prohibitions, but to arrive at
conventions.*" (R. Carnap)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[David Mertz]

On Oct 23, 2016 9:12 AM, "Danilo J. S. Bellini" 
wrote:
Actually, itertools.accumulate and functools.reduce have their parameters
reversed, and accumulate doesn't have a "start" parameter.

def accumulate2(fn=operator.add, it, start=None):
if start is not None:
 it = iterations.chain([start], it)
return itertools.accumulate(it, fn)

I would have preferred this signature to start with, but it's easy to wrap.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Danilo J. S. Bellini]

>
> Ah, that makes SIX existing solutions. Do we need a seventh?

It might have dozens of solutions, perhaps an infinity of solutions.
Brainfuck and assembly can be used, or even turing machine instructions...

But there should be one, and preferably only one, OBVIOUS way to do it.
Readability counts.

Reduce lost the built-in status on Python 3. Lambdas lost the decomposing
arguments like "lambda (a, b), c: a + b * c". Using a for loop section
inside a generator expression or list/set/dict comprehension allow the
decomposing arguments and doesn't need a function to be imported. Actually,
itertools.accumulate and functools.reduce have their parameters reversed,
and accumulate doesn't have a "start" parameter.

Actually, the example I give in this thread is about a fold/reduce trying
to show it's way simpler than the other solutions. I didn't paste here any
scan use case because I sent links with several use cases, should I paste
their contents here? The PyScanPrev link (https://github.com/
danilobellini/pyscanprev) has several use case examples (including some
just for a comparison with other possible solutions) and even have a full
rationale for this idea.

-- 
Danilo J. S. Bellini
---
"*It is not our business to set up prohibitions, but to arrive at
conventions.*" (R. Carnap)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Steven D'Aprano]

On Sun, Oct 23, 2016 at 12:57:10PM -0200, Danilo J. S. Bellini wrote:
> The idea is to let generator expressions and list/set comprehensions have a
> clean syntax to access its last output. That would allow them to be an
> alternative syntax to the scan higher-order function [1] (today implemented
> in the itertools.accumulate function), which leads to an alternative way to
> write a fold/reduce. It would be nice to have something like:

[cut suggested syntax]

> instead of a reduce:

[cut four existing ways to solve the problem]

Why do we need a FIFTH way to solve this problem? What you are 
describing is *exactly* the use case for a reduce or fold function. Why 
add special magic syntax to make comprehensions do even more?

Not everything needs to be a one liner. It's okay to import reduce to do 
a reduce. Its okay to write an actual for-loop.

> Actually, I already wrote a solution for something similar to that:
> PyScanPrev [2]. 

Ah, that makes SIX existing solutions. Do we need a seventh?



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Danilo J. S. Bellini]

>
> Of course. But if you want last(), why not just spell the utility function
> as I did? [...]
>
I'm not against a general "last", I just said the main idea of this thread
is the access to the previous iteration output in a list/set/dict
comprehension or generator expression.

> Actually, your code is similar to the reference implementation I wrote for
PyScanPrev, the main difference is that my "last" raises a StopIteration on
an empty input instead of an UnboundLocalError:
https://github.com/danilobellini/pyscanprev/blob/v0.1.0/pyscanprev.py#L148
When the input is a sequence, it should be optimized to get the item at the
index -1.

That works fine for any iteratable (including a list, array, etc), whether
> or not it's a reduction/accumulation.
>
Lists and arrays don't need to be traversed.

Consuming the iterator is *necessary* to get the last item. There's no way
> around that.
>
Not if there's enough information to create the last value. Perhaps on the
it = iter(range(999)) one can get 2 values (call next(it) twice) and
use its __length_hint__ to create the last value. But I think only
sequences should have such an optimization, not iterators.

-- 
Danilo J. S. Bellini
---
"*It is not our business to set up prohibitions, but to arrive at
conventions.*" (R. Carnap)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[David Mertz]

Consuming the iterator is *necessary* to get the last item. There's no way
around that.

Obviously, you could itertools.tee() it first if you don't mind the cache
space. But there cannot be a generic "jump to the end" of an iterator
without being destructive.

On Oct 23, 2016 8:43 AM, "Steven D'Aprano"  wrote:

> On Sun, Oct 23, 2016 at 08:37:07AM -0700, David Mertz wrote:
> > Of course. But if you want last(), why not just spell the utility
> function
> > as I did? I.e. as a function:
> >
> > def last(it):
> >  for item in it:
> >  pass
> > return item
> >
> > That works fine for any iteratable (including a list, array, etc),
> whether
> > or not it's a reduction/accumulation.
>
> That's no good, because it consumes the iterator. Yes, you get
> the last value, but you actually needed to do work on all the
> previous values too.
>
>
> --
> Steve
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Steven D'Aprano]

On Sun, Oct 23, 2016 at 08:37:07AM -0700, David Mertz wrote:
> Of course. But if you want last(), why not just spell the utility function
> as I did? I.e. as a function:
> 
> def last(it):
>  for item in it:
>  pass
> return item
> 
> That works fine for any iteratable (including a list, array, etc), whether
> or not it's a reduction/accumulation.

That's no good, because it consumes the iterator. Yes, you get 
the last value, but you actually needed to do work on all the 
previous values too.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Easily remove characters from a string.

[Steven D'Aprano]

On Sat, Oct 22, 2016 at 03:34:23PM +0700, Simon Mark Holland wrote:
> Having researched this as heavily as I am capable with limited experience,
> I would like to suggest a Python 3 equivalent to string.translate() that
> doesn't require a table as input.  Maybe in the form of str.stripall() or
> str.replaceall().

stripall() would not be appropriate: "strip" refers to removing from the 
front and end of the string, not the middle, and str.strip() already 
implements a "strip all" functionality:

py> '+--+*abcd+-*xyz-*+-'.strip('*+-')
'abcd+-*xyz'


But instead of a new method, why not fix translate() to be more user- 
friendly? Currently, it takes two method calls to delete characters 
using translate:

table = str.maketrans('', '', '*+-.!?')
newstring = mystring.translate(table)

That's appropriate when you have a big translation table which you are 
intending to use many times, but its a bit clunky for single, one-off 
uses.

Maybe we could change the API of translate to something like this:

def translate(self, *args):
if len(args) == 1:
# Same as the existing behaviour.
table = args[0]
elif len(args) == 3:
table = type(self).maketrans(*args)
else:
raise TypeError('too many or not enough arguments')
...


Then we could write:

newstring = mystring.translate('', '', '1234567890')

to delete the digits. 

So we could fix this... but should we? Is this *actually* a problem that 
needs fixing, or are we just adding unnecessary complexity?


> My reasoning is that while it is currently possible to easily strip()
> preceding and trailing characters, and even replace() individual characters
> from a string, 

Stripping from the front and back is a very common operation; in my 
experience, replacing is probably half as common, maybe even less. But 
deleting is even less common.


> My proposal is that if strip() and replace() are important enough to
> receive modules, then the arguably more common operation (in terms of
> programming tutorials, if not mainstream development) of just removing all
> instances of specified numbers, punctuation, or even letters etc from a
> list of characters should also.

I think the reason that deleting characters is common in tutorials is 
that it is a simple, easy, obvious task that can be programmed by a 
beginner in just a few lines. I don't think it is actually something 
that people need to do very often, outside of exercises.



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[David Mertz]

Of course. But if you want last(), why not just spell the utility function
as I did? I.e. as a function:

def last(it):
 for item in it:
 pass
return item

That works fine for any iteratable (including a list, array, etc), whether
or not it's a reduction/accumulation.

On Oct 23, 2016 8:29 AM, "Danilo J. S. Bellini" 
wrote:

> What is `last(inf_iter)`. E.g `last(count())`.
>
> The "last" is just a helper function that gets the last value of an
> iterable. On sequences, it can be written to get the item at index -1 to
> avoid traversing it. Using it on endless iterables makes no sense.
>
> This makes it clear that is the users job to make sure `it` terminates.
>
> If one call "last" for something that doesn't terminate, an "endless"
> iterable, well, it's pretty obvious that it won't "end" nicely. It's not
> the Python job to solve the Entscheidungsproblem. If you call "sorted" on
> endless iterables, it would behave like "last", doesn't it?
>
> The whole point of this idea is the scan as a generator expression or
> list/set comprehension that can access the previous iteration output.
> Reduce/fold is just the last value of a scan, and the scan is still defined
> when there's no "last value".
>
> --
> Danilo J. S. Bellini
> ---
> "*It is not our business to set up prohibitions, but to arrive at
> conventions.*" (R. Carnap)
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Danilo J. S. Bellini]

>
> What is `last(inf_iter)`. E.g `last(count())`.

The "last" is just a helper function that gets the last value of an
iterable. On sequences, it can be written to get the item at index -1 to
avoid traversing it. Using it on endless iterables makes no sense.

This makes it clear that is the users job to make sure `it` terminates.

If one call "last" for something that doesn't terminate, an "endless"
iterable, well, it's pretty obvious that it won't "end" nicely. It's not
the Python job to solve the Entscheidungsproblem. If you call "sorted" on
endless iterables, it would behave like "last", doesn't it?

The whole point of this idea is the scan as a generator expression or
list/set comprehension that can access the previous iteration output.
Reduce/fold is just the last value of a scan, and the scan is still defined
when there's no "last value".

-- 
Danilo J. S. Bellini
---
"*It is not our business to set up prohibitions, but to arrive at
conventions.*" (R. Carnap)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[David Mertz]

What is `last(inf_iter)`. E.g `last(count())`.

To me, the obvious spelling is:

for last in it: pass
doSomething(last)

This makes it clear that is the users job to make sure `it` terminates.
There's no general way to get the last item without looking through all the
earlier ones.

On Oct 23, 2016 7:58 AM, "Danilo J. S. Bellini" 
wrote:

> The idea is to let generator expressions and list/set comprehensions have
> a clean syntax to access its last output. That would allow them to be an
> alternative syntax to the scan higher-order function [1] (today implemented
> in the itertools.accumulate function), which leads to an alternative way
> to write a fold/reduce. It would be nice to have something like:
>
> >>> last(abs(prev - x) for x in [3, 4, 5] from prev = 2)
> 2
>
> instead of a reduce:
>
> >>> from functools import reduce
> >>> reduce(lambda prev, x: abs(prev - x), [3, 4, 5], 2)
> 2
>
> or an imperative approach:
>
> >>> prev = 2
> >>> for x in [3, 4, 5]:
> ... prev = abs(prev - x)
> >>> prev
> 2
>
> or getting the last from accumulate:
>
> >>> from itertools import accumulate
> >>> list(accumulate([2, 3, 4, 5], lambda prev, x: abs(prev - x)))[-1]
> 2
>
> or...
>
> >>> [prev for prev in [2]
> ...   for x in [3, 4, 5]
> ...   for prev in [abs(prev - x)]
> ... ][-1]
> 2
>
> Actually, I already wrote a solution for something similar to that:
> PyScanPrev [2]. I'm using bytecode manipulation to modify the generator
> expression and set/list comprehensions semantics to create a "scan", but
> it has the limitation of using only code with a valid syntax as the input,
> so I can't use "from" inside a generator expression / list comprehension.
> The solution was to put the first output into the iterable and define the
> "prev" name elsewhere:
>
> >>> last(abs(prev - x) for x in [2, 3, 4, 5])
> 2
>
> That line works with PyScanPrev (on Python 3.4 and 3.5) when defined in a
> function with a @enable_scan("prev") decorator. That was enough to create
> a "test suite" of doctest-based examples that shows several scan use cases
> [2].
>
> This discussion started in a Brazilian list when someone asked how she
> could solve a simple uppercase/lowercase problem [3]. The goal was to
> alternate the upper/lower case of a string while neglecting the chars that
> doesn't apply (i.e., to "keep the state" when the char isn't a letter). After
> the discussion, I wrote the PyScanPrev package, and recently I've added
> this historical "alternate" function as the "conditional toggling"
> example [4].
>
> Then I ask, can Python include that "scan" access to the last output in
> its list/set/dict comprehension and generator expression syntax? There are
> several possible applications for the scan itself as well as for the
> fold/reduce (signal processing, control theory, physics, economics, etc.),
> some of them I included as PyScanPrev examples. Some friends (people who
> like control engineering and/or signal processing) liked the "State-space
> model" example, where I included a "leaking bucket-spring-damper"
> simulation using the scan-enabled generator expressions [5].
>
> About the syntax, there are several ideas on how that can be written.
> Given a "prev" identifier, a "target" identifier, an input "iterable" and
> an optional "start" value (and perhaps an optional "echo_start", which I
> assume True by default), some of them are:
>
> [func(prev, target) for target in iterable from prev = start]
> [func(prev, target) for target in iterable] -> prev = start
> [func(prev, target) for target in iterable] -> prev as start
> [func(prev, target) for target in iterable] from prev = start
> [func(prev, target) for target in iterable] from prev as start
> [func(prev, target) for target in iterable] with prev as start
> prev = start -> [func(prev, target) for target in iterable]
> prev(start) -> [func(prev, target) for target in iterable]
> [func(prev, target) for prev -> target in start -> iterable]
> [prev = start -> func(prev, target) for target in iterable]
>
> # With ``start`` being the first value of the iterable, i.e.,
> #   iterable = prepend(start, data)
> [func(prev, target) for target in iterable from prev]
> [func(prev, target) for target in iterable] -> prev
> [func(prev, target) for target in iterable] from prev
> prev -> [func(prev, target) for target in iterable]
>
> Before writing PyScanPrev, in [6] (Brazilian Portuguese) I used stackfull
> [7] to implement that idea, an accumulator example using that library is:
>
> >>> from stackfull import push, pop, stack
> >>> [push(pop() + el if stack() else el) for el in range(5)]
> [0, 1, 3, 6, 10]
> >>> list(itertools.accumulate(range(5)))
> [0, 1, 3, 6, 10]
>
> There are more I can say (e.g. the pyscanprev.scan function has a "start"
> value and an "echo_start" keyword argument, resources I missed in
> itertools.accumulate) but the links below already have a lot of information.
>
> [1] 

[Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Danilo J. S. Bellini]

The idea is to let generator expressions and list/set comprehensions have a
clean syntax to access its last output. That would allow them to be an
alternative syntax to the scan higher-order function [1] (today implemented
in the itertools.accumulate function), which leads to an alternative way to
write a fold/reduce. It would be nice to have something like:

>>> last(abs(prev - x) for x in [3, 4, 5] from prev = 2)
2

instead of a reduce:

>>> from functools import reduce
>>> reduce(lambda prev, x: abs(prev - x), [3, 4, 5], 2)
2

or an imperative approach:

>>> prev = 2
>>> for x in [3, 4, 5]:
... prev = abs(prev - x)
>>> prev
2

or getting the last from accumulate:

>>> from itertools import accumulate
>>> list(accumulate([2, 3, 4, 5], lambda prev, x: abs(prev - x)))[-1]
2

or...

>>> [prev for prev in [2]
...   for x in [3, 4, 5]
...   for prev in [abs(prev - x)]
... ][-1]
2

Actually, I already wrote a solution for something similar to that:
PyScanPrev [2]. I'm using bytecode manipulation to modify the generator
expression and set/list comprehensions semantics to create a "scan", but it
has the limitation of using only code with a valid syntax as the input, so
I can't use "from" inside a generator expression / list comprehension. The
solution was to put the first output into the iterable and define the
"prev" name elsewhere:

>>> last(abs(prev - x) for x in [2, 3, 4, 5])
2

That line works with PyScanPrev (on Python 3.4 and 3.5) when defined in a
function with a @enable_scan("prev") decorator. That was enough to create a
"test suite" of doctest-based examples that shows several scan use cases
[2].

This discussion started in a Brazilian list when someone asked how she
could solve a simple uppercase/lowercase problem [3]. The goal was to
alternate the upper/lower case of a string while neglecting the chars that
doesn't apply (i.e., to "keep the state" when the char isn't a letter). After
the discussion, I wrote the PyScanPrev package, and recently I've added
this historical "alternate" function as the "conditional toggling" example
[4].

Then I ask, can Python include that "scan" access to the last output in its
list/set/dict comprehension and generator expression syntax? There are
several possible applications for the scan itself as well as for the
fold/reduce (signal processing, control theory, physics, economics, etc.),
some of them I included as PyScanPrev examples. Some friends (people who
like control engineering and/or signal processing) liked the "State-space
model" example, where I included a "leaking bucket-spring-damper"
simulation using the scan-enabled generator expressions [5].

About the syntax, there are several ideas on how that can be written. Given
a "prev" identifier, a "target" identifier, an input "iterable" and an
optional "start" value (and perhaps an optional "echo_start", which I
assume True by default), some of them are:

[func(prev, target) for target in iterable from prev = start]
[func(prev, target) for target in iterable] -> prev = start
[func(prev, target) for target in iterable] -> prev as start
[func(prev, target) for target in iterable] from prev = start
[func(prev, target) for target in iterable] from prev as start
[func(prev, target) for target in iterable] with prev as start
prev = start -> [func(prev, target) for target in iterable]
prev(start) -> [func(prev, target) for target in iterable]
[func(prev, target) for prev -> target in start -> iterable]
[prev = start -> func(prev, target) for target in iterable]

# With ``start`` being the first value of the iterable, i.e.,
#   iterable = prepend(start, data)
[func(prev, target) for target in iterable from prev]
[func(prev, target) for target in iterable] -> prev
[func(prev, target) for target in iterable] from prev
prev -> [func(prev, target) for target in iterable]

Before writing PyScanPrev, in [6] (Brazilian Portuguese) I used stackfull
[7] to implement that idea, an accumulator example using that library is:

>>> from stackfull import push, pop, stack
>>> [push(pop() + el if stack() else el) for el in range(5)]
[0, 1, 3, 6, 10]
>>> list(itertools.accumulate(range(5)))
[0, 1, 3, 6, 10]

There are more I can say (e.g. the pyscanprev.scan function has a "start"
value and an "echo_start" keyword argument, resources I missed in
itertools.accumulate) but the links below already have a lot of information.

[1] https://en.wikipedia.org/wiki/Prefix_sum#Scan_higher_order_function
[2] https://pypi.python.org/pypi/pyscanprev
[3] https://groups.google.com/forum/#!topic/grupy-sp/wTIj6G5_5S0
[4]
https://github.com/danilobellini/pyscanprev/blob/v0.1.0/examples/conditional-toggling.rst
[5]
https://github.com/danilobellini/pyscanprev/blob/v0.1.0/examples/state-space.rst
[6] https://groups.google.com/forum/#!topic/grupy-sp/UZp-lVSWK1s
[7] https://pypi.python.org/pypi/stackfull

-- 
Danilo J. S. Bellini
---
"*It is not our business to set up prohibitions, but to arrive at
conventions.*" (R. Carnap)

Re: [Python-ideas] Easily remove characters from a string.

[M.-A. Lemburg]

On 22.10.2016 10:34, Simon Mark Holland wrote:
> Having researched this as heavily as I am capable with limited experience,
> I would like to suggest a Python 3 equivalent to string.translate() that
> doesn't require a table as input.  Maybe in the form of str.stripall() or
> str.replaceall().
> 
> My reasoning is that while it is currently possible to easily strip()
> preceding and trailing characters, and even replace() individual characters
> from a string, to replace more than one characters from anywhere within the
> string requires (i believe) at its simplest a command like this :
> 
> some_string.translate(str.maketrans('','','0123456789'))
> 
> In Python 2.* however we could say ...
> 
> some_string.translate(None, '0123456789')
> 
> My proposal is that if strip() and replace() are important enough to
> receive modules, then the arguably more common operation (in terms of
> programming tutorials, if not mainstream development) of just removing all
> instances of specified numbers, punctuation, or even letters etc from a
> list of characters should also.
> 
> I wholeheartedly admit that there are MANY other ways to do this (including
> RegEx and List Comprehensions), as listed in the StackOverflow answer
> below.   However the same could be said for replace() and strip().
> http://stackoverflow.com/questions/22187233/how-to-delete-all-instances-of-a-character-in-a-string-in-python
> 
> This is my first suggestion and welcome any and all feedback, even if this
> is a silly idea I would really like to know why it is.  I have not seen
> discussion of this before, but if there is such a discussion I would
> welcome being directed to it.

Could you perhaps give a use case for what you have in mind ?

I usually go straight to the re module for anything that's
non-trivial in terms of string manipulation, or use my
mxTextTools for more complex stuff.

re.sub() would be the natural choice for replacing multiple
chars or removing multiple chars in one go.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Oct 23 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/