Re: Scraping of Google Map to get the address of locations for given geocodes.
On 2021-09-02 at 20:12:19 -0300, Michio Suginoo wrote: > I have the following question regarding how to scrape Google Map to get > address based on a given list of geocodes. > Given a list of geocodes (latitude; longitude) of locations, I would like > to scrape municipalities of all the spots in the list. How can I do that? > > For example, I have > > lat lon > > 1) -34.5722317 -58.4314464 > > 2) -34.553906 -58.4520949 > > 3) -34.5661444 -58.4964289 > > 4) -34.5648053 -58.4431567 > > 5) -34.5664089 -58.4323004 > > 6) -34.5664089 -58.4323004 > > > > And I want to get the name of municipality (city/town), or alternatively > the address, of each location. > > > I would appreciate it, if anyone can advise me how to do that. Is scraping Google Map a hard requirement? Both geopy¹ and geocoder² can translate those coordinates into street addresses directly, and their documentation even explains how to bypass the libraries and go straight to the service providers. ¹ https://geopy.readthedocs.io/en/latest/ ² https://geocoder.readthedocs.io/ -- https://mail.python.org/mailman/listinfo/python-list
Scraping of Google Map to get the address of locations for given geocodes.
Dear all, I have the following question regarding how to scrape Google Map to get address based on a given list of geocodes. Given a list of geocodes (latitude; longitude) of locations, I would like to scrape municipalities of all the spots in the list. How can I do that? For example, I have lat lon 1) -34.5722317 -58.4314464 2) -34.553906 -58.4520949 3) -34.5661444 -58.4964289 4) -34.5648053 -58.4431567 5) -34.5664089 -58.4323004 6) -34.5664089 -58.4323004 And I want to get the name of municipality (city/town), or alternatively the address, of each location. I would appreciate it, if anyone can advise me how to do that. Thanks Best regards Michio -- https://mail.python.org/mailman/listinfo/python-list
Re: What is not working with my "map" usage?
On Sat, Sep 22, 2018, Victor (vhnguy...@yahoo.com) wrote Let me use a different input args and display them below. Basically, I am hoping to add up all elements of each nested list. So at first it should start with [1,11,111] ==> 1+11+111 = 123. But instead, it appears to take the 1st element from each nested list to add up [1,2,3] = 6. How should it be corrected? Thx. > >> alist = [[1,11,111], [2,22,222], [3,33,333]] > >> list(map(add_all_elements,*alist) > > Trylist(map(add_all_elements, alist)) instead. If you give a map a single list as a second argument, it will visit eachelement of that single list (which in this case are the sublists you want). But that * is replacing the single list with the list contents as separate arguments,so you are literally telling map to visit all three sublists in parallel,and it does exactly what you told it to. The * and ** prefixes really should be avoided as much as possible,unless you know exactly what you are getting out of them. Roger ChristmanPennsylvania State University -- https://mail.python.org/mailman/listinfo/python-list
Re: What is not working with my "map" usage?
On Saturday, September 22, 2018 at 12:20:08 PM UTC-7, Thomas Jollans wrote: > On 22/09/2018 20:18, Victor via Python-list wrote: > > On Saturday, September 22, 2018 at 6:22:32 AM UTC-7, Peter Otten wrote: > >> Victor via Python-list wrote: > >> > >>> Let me use a different input args and display them below. Basically, I am > >>> hoping to add up all elements of each nested list. So at first it should > >>> start with [1,11,111] ==> 1+11+111 = 123. But instead, it appears to take > >>> the 1st element from each nested list to add up [1,2,3] = 6. How should > >>> it be corrected? Thx. > >> > >> I see three options. You can > >> > >> (1) use a list comprehension > >> > >> [add_all_elements(*sub) for sub in alist] > >> > >> (2) replace map() with itertools.starmap() > >> > >> list(itertools.starmap(add_all_elements, alist)) > >> > >> (3) change your function's signature from add_all_elements(*args) to > >> add_all_elements(args), either by modifying it directly or by wrapping it > >> into another function > >> > >> list(map(lambda args: add_all_elements(*args), alist)) > >> > >> (3a) My personal preference would be to change the signature and then use > >> the list comprehension > >> > >> def add_all_elements(args): ... > >> [add_all_elements(sub) for sub in alist] > > > > Hi Peter, > > Thank you for your suggested solutions. They all work. But I just want to > > know what is wrong with my doing: > > > > list(map(add_all_elements,*alist)) > > > > Theoretically, each list element is passed to add_all_elements. And if my > > alist is [[1, 11, 111], [2, 22, 222], [3, 33, 333]], then the 1st list > > element must be this [1,11,111] passed as args into add_all_elements. > > Now, > >alist = [[1,11,111], [2,22,222], [3,33,333]] > > so `map(add_all_alements, *alist)` is equivalent to > >map(add_all_elements, >[1,11,111], >[2,22,222], >[3,33,333]) > > According to the docs [1], map(function, iterable, ...) > > "applies function to every item of iterable, yielding the results. If > additional iterable arguments are passed, function must take that many > arguments and is applied to the items from all iterables in parallel." > > So map takes the first item(s) of the argument(s), and applies the > function to them, followed by the second item(s), and so on. > > In other words: > > def map(function, *iterables): > for args in zip(iterables): > yield function(*args) > > > [1] https://docs.python.org/3/library/functions.html#map > > Come to think of it, this suggests a rather silly alternative "solution": > > map(add_all_elements, *zip(*alist)) > > > > > In other words, the following should have happened: > > > >>>> add_all_elements (*[1,11,111]) > > My args = (1, 11, 111) > > > > i = 1 > > BEFORE total = 0 > > AFTER total = 1 > > > > i = 11 > > BEFORE total = 1 > > AFTER total = 12 > > > > i = 111 > > BEFORE total = 12 > > AFTER total = 123 > > > > FINAL total = 123 > > > > 123 > > > > Again, thanks! > > Thanks all, I got it now! -- https://mail.python.org/mailman/listinfo/python-list
Re: What is not working with my "map" usage?
On 22/09/2018 20:18, Victor via Python-list wrote: On Saturday, September 22, 2018 at 6:22:32 AM UTC-7, Peter Otten wrote: Victor via Python-list wrote: Let me use a different input args and display them below. Basically, I am hoping to add up all elements of each nested list. So at first it should start with [1,11,111] ==> 1+11+111 = 123. But instead, it appears to take the 1st element from each nested list to add up [1,2,3] = 6. How should it be corrected? Thx. I see three options. You can (1) use a list comprehension [add_all_elements(*sub) for sub in alist] (2) replace map() with itertools.starmap() list(itertools.starmap(add_all_elements, alist)) (3) change your function's signature from add_all_elements(*args) to add_all_elements(args), either by modifying it directly or by wrapping it into another function list(map(lambda args: add_all_elements(*args), alist)) (3a) My personal preference would be to change the signature and then use the list comprehension def add_all_elements(args): ... [add_all_elements(sub) for sub in alist] Hi Peter, Thank you for your suggested solutions. They all work. But I just want to know what is wrong with my doing: list(map(add_all_elements,*alist)) Theoretically, each list element is passed to add_all_elements. And if my alist is [[1, 11, 111], [2, 22, 222], [3, 33, 333]], then the 1st list element must be this [1,11,111] passed as args into add_all_elements. Now, alist = [[1,11,111], [2,22,222], [3,33,333]] so `map(add_all_alements, *alist)` is equivalent to map(add_all_elements, [1,11,111], [2,22,222], [3,33,333]) According to the docs [1], map(function, iterable, ...) "applies function to every item of iterable, yielding the results. If additional iterable arguments are passed, function must take that many arguments and is applied to the items from all iterables in parallel." So map takes the first item(s) of the argument(s), and applies the function to them, followed by the second item(s), and so on. In other words: def map(function, *iterables): for args in zip(iterables): yield function(*args) [1] https://docs.python.org/3/library/functions.html#map Come to think of it, this suggests a rather silly alternative "solution": map(add_all_elements, *zip(*alist)) In other words, the following should have happened: add_all_elements (*[1,11,111]) My args = (1, 11, 111) i = 1 BEFORE total = 0 AFTER total = 1 i = 11 BEFORE total = 1 AFTER total = 12 i = 111 BEFORE total = 12 AFTER total = 123 FINAL total = 123 123 Again, thanks! -- https://mail.python.org/mailman/listinfo/python-list
Re: What is not working with my "map" usage?
Victor via Python-list wrote: > On Saturday, September 22, 2018 at 6:22:32 AM UTC-7, Peter Otten wrote: >> Victor via Python-list wrote: >> >> > Let me use a different input args and display them below. Basically, I >> > am >> > hoping to add up all elements of each nested list. So at first it >> > should >> > start with [1,11,111] ==> 1+11+111 = 123. But instead, it appears to >> > take >> > the 1st element from each nested list to add up [1,2,3] = 6. How >> > should >> > it be corrected? Thx. >> >> I see three options. You can >> >> (1) use a list comprehension >> >> [add_all_elements(*sub) for sub in alist] >> >> (2) replace map() with itertools.starmap() >> >> list(itertools.starmap(add_all_elements, alist)) >> >> (3) change your function's signature from add_all_elements(*args) to >> add_all_elements(args), either by modifying it directly or by wrapping it >> into another function >> >> list(map(lambda args: add_all_elements(*args), alist)) >> >> (3a) My personal preference would be to change the signature and then use >> the list comprehension >> >> def add_all_elements(args): ... >> [add_all_elements(sub) for sub in alist] > > Hi Peter, > Thank you for your suggested solutions. They all work. But I just want > to know what is wrong with my doing: > > list(map(add_all_elements,*alist)) > > Theoretically, each list element is passed to add_all_elements. And if my > alist is [[1, 11, 111], [2, 22, 222], [3, 33, 333]], then the 1st list > element must be this [1,11,111] passed as args into add_all_elements. > > In other words, the following should have happened: > >>>> add_all_elements (*[1,11,111]) That's not what happens. Try it with a function that passes through its arguments unchanged: >>> items = [[1, 11, 111], [2, 22, 222], [3, 33, 333]] >>> list(map(lambda *args: args, *items)) [(1, 2, 3), (11, 22, 33), (111, 222, 333)] The star before items is evaluated directly rather than magically passed on to the call of add_all_elements. map(f, *items) is equivalent to map(f, items[0], items[1], items[2]) which calls f with f(items[0][0], items[1][0], items[2][0]) f(items[0][1], items[1][1], items[2][1]) f(items[0][2], items[1][2], items[2][2]) If you think of your list of lists as a matrix that matrix is effectively transposed (x and y axis are swapped). >>> list(map(lambda *args: sum(args), *items)) [6, 66, 666] In principle you could transpose the matrix twice >>> list(map(lambda *args: sum(args), *zip(*items))) [123, 246, 369] but that's not a very efficient approach. Another gotcha to be aware of is that map() (and zip()) stop when the shortest argument ends: >>> list(map(lambda *args: sum(args), [], *items)) [] >>> list(map(lambda *args: sum(args), [42], *items)) [48] >>> list(map(lambda *args: sum(args), [42, 42], *items)) [48, 108] >>> list(map(lambda *args: sum(args), [42, 42, 42, 42, 42], *items)) [48, 108, 708] -- https://mail.python.org/mailman/listinfo/python-list
Re: What is not working with my "map" usage?
On Saturday, September 22, 2018 at 6:22:32 AM UTC-7, Peter Otten wrote: > Victor via Python-list wrote: > > > Let me use a different input args and display them below. Basically, I am > > hoping to add up all elements of each nested list. So at first it should > > start with [1,11,111] ==> 1+11+111 = 123. But instead, it appears to take > > the 1st element from each nested list to add up [1,2,3] = 6. How should > > it be corrected? Thx. > > I see three options. You can > > (1) use a list comprehension > > [add_all_elements(*sub) for sub in alist] > > (2) replace map() with itertools.starmap() > > list(itertools.starmap(add_all_elements, alist)) > > (3) change your function's signature from add_all_elements(*args) to > add_all_elements(args), either by modifying it directly or by wrapping it > into another function > > list(map(lambda args: add_all_elements(*args), alist)) > > (3a) My personal preference would be to change the signature and then use > the list comprehension > > def add_all_elements(args): ... > [add_all_elements(sub) for sub in alist] Hi Peter, Thank you for your suggested solutions. They all work. But I just want to know what is wrong with my doing: list(map(add_all_elements,*alist)) Theoretically, each list element is passed to add_all_elements. And if my alist is [[1, 11, 111], [2, 22, 222], [3, 33, 333]], then the 1st list element must be this [1,11,111] passed as args into add_all_elements. In other words, the following should have happened: >>> add_all_elements (*[1,11,111]) My args = (1, 11, 111) i = 1 BEFORE total = 0 AFTER total = 1 i = 11 BEFORE total = 1 AFTER total = 12 i = 111 BEFORE total = 12 AFTER total = 123 FINAL total = 123 123 Again, thanks! -- https://mail.python.org/mailman/listinfo/python-list
Re: What is not working with my "map" usage?
Victor via Python-list wrote: > Let me use a different input args and display them below. Basically, I am > hoping to add up all elements of each nested list. So at first it should > start with [1,11,111] ==> 1+11+111 = 123. But instead, it appears to take > the 1st element from each nested list to add up [1,2,3] = 6. How should > it be corrected? Thx. I see three options. You can (1) use a list comprehension [add_all_elements(*sub) for sub in alist] (2) replace map() with itertools.starmap() list(itertools.starmap(add_all_elements, alist)) (3) change your function's signature from add_all_elements(*args) to add_all_elements(args), either by modifying it directly or by wrapping it into another function list(map(lambda args: add_all_elements(*args), alist)) (3a) My personal preference would be to change the signature and then use the list comprehension def add_all_elements(args): ... [add_all_elements(sub) for sub in alist] -- https://mail.python.org/mailman/listinfo/python-list
Re: What is not working with my "map" usage?
Let me use a different input args and display them below. Basically, I am hoping to add up all elements of each nested list. So at first it should start with [1,11,111] ==> 1+11+111 = 123. But instead, it appears to take the 1st element from each nested list to add up [1,2,3] = 6. How should it be corrected? Thx. >>> alist = [[1,11,111], [2,22,222], [3,33,333]] >>> list(map(add_all_elements,*alist)) My args = (1, 2, 3) i = 1 BEFORE total = 0 AFTER total = 1 i = 2 BEFORE total = 1 AFTER total = 3 i = 3 BEFORE total = 3 AFTER total = 6 FINAL total = 6 My args = (11, 22, 33) i = 11 BEFORE total = 0 AFTER total = 11 i = 22 BEFORE total = 11 AFTER total = 33 i = 33 BEFORE total = 33 AFTER total = 66 FINAL total = 66 My args = (111, 222, 333) i = 111 BEFORE total = 0 AFTER total = 111 i = 222 BEFORE total = 111 AFTER total = 333 i = 333 BEFORE total = 333 AFTER total = 666 FINAL total = 666 [6, 66, 666] -- https://mail.python.org/mailman/listinfo/python-list
Re: What is not working with my "map" usage?
On 21/09/2018 23:29, Viet Nguyen via Python-list wrote:
Hi,
I want to add up all of the list elements. But when I use the "map" function, it didn't
seem to work as I expect. Could someone point out how "map" can be applied here then?
def add_all_elements (*args):
total = 0
for i in args:
print(type(i))
print("i = %s" % i)
print("BEFORE total = %s" % total)
total += int(i)
print("AFTER total = %s\n" % total)
print("FINAL total = %s\n" % total)
return total
alist = ['2', '09', '49']
## this one works Okay
add_all_elements(*alist)
> [...]
list(map(add_all_elements,alist))
Read the output in both cases and try to figure out what 'args' is. Or
you could print out 'args' at the top of your function to lend you a
hand. That should help you understand what the difference between the
two cases is, and what map() is doing.
Good luck!
Thomas
--
https://mail.python.org/mailman/listinfo/python-list
Re: What is not working with my "map" usage?
Hi Viet,
map applies the function to each of the elements of the list you provide.
It would be roughly equivalent to:
[add_all_elements(x) for x in alist]
It may help you to consider the term and function "map" from the view of linear
algebra.
Apparently it's a common term:
https://en.wikipedia.org/wiki/Map_%28higher-order_function%29?wprov=sfla1
HTH
On September 21, 2018 11:29:41 PM GMT+02:00, Viet Nguyen via Python-list
wrote:
>Hi,
>
>I want to add up all of the list elements. But when I use the "map"
>function, it didn't seem to work as I expect. Could someone point out
>how "map" can be applied here then?
>
>def add_all_elements (*args):
>total = 0
>for i in args:
> print(type(i))
> print("i = %s" % i)
> print("BEFORE total = %s" % total)
> total += int(i)
> print("AFTER total = %s\n" % total)
>print("FINAL total = %s\n" % total)
>return total
>
>
>alist = ['2', '09', '49']
>
>
>## this one works Okay
>
>add_all_elements(*alist)
>
>i = 2
>BEFORE total = 0
>AFTER total = 2
>
>
>i = 09
>BEFORE total = 2
>AFTER total = 11
>
>
>i = 49
>BEFORE total = 11
>AFTER total = 60
>
>FINAL total = 60
>
>
>## Why is this NOT Okay when I use map ?? What must I change ?
>
>>>> list(map(add_all_elements,alist))
>
>i = 2
>BEFORE total = 0
>AFTER total = 2
>
>FINAL total = 2
>
>
>i = 09
>BEFORE total = 0
>AFTER total = 9
>
>FINAL total = 9
>
>
>i = 49
>BEFORE total = 0
>AFTER total = 49
>
>FINAL total = 49
>
>[2, 9, 49]
>
>
>Thanks,
>Viet
>--
>https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
What is not working with my "map" usage?
Hi,
I want to add up all of the list elements. But when I use the "map" function,
it didn't seem to work as I expect. Could someone point out how "map" can be
applied here then?
def add_all_elements (*args):
total = 0
for i in args:
print(type(i))
print("i = %s" % i)
print("BEFORE total = %s" % total)
total += int(i)
print("AFTER total = %s\n" % total)
print("FINAL total = %s\n" % total)
return total
alist = ['2', '09', '49']
## this one works Okay
add_all_elements(*alist)
i = 2
BEFORE total = 0
AFTER total = 2
i = 09
BEFORE total = 2
AFTER total = 11
i = 49
BEFORE total = 11
AFTER total = 60
FINAL total = 60
## Why is this NOT Okay when I use map ?? What must I change ?
>>> list(map(add_all_elements,alist))
i = 2
BEFORE total = 0
AFTER total = 2
FINAL total = 2
i = 09
BEFORE total = 0
AFTER total = 9
FINAL total = 9
i = 49
BEFORE total = 0
AFTER total = 49
FINAL total = 49
[2, 9, 49]
Thanks,
Viet
--
https://mail.python.org/mailman/listinfo/python-list
in multiprocessing pool map how to stop one process from executing ahead
I am excecting custom commands like shell on multiple linux hosts. and if in one host one of the commands fail. I want that process not to proceed. If the remote command throws an error i am logging it .. but the process goes to next command . but if i terminate the command, the process will terminate all other processes on other linux machines as well. -- https://mail.python.org/mailman/listinfo/python-list
Re: Plot map wit a white and black box
On Monday, January 8, 2018 at 1:16:08 PM UTC, jorge@cptec.inpe.br wrote: > Hi, > > Please, I woudl like to plot a map like this figure. How can I do this > using Python2.7 > > Thanks, > > Conrado Figures don't get through and you've all ready asked this question, possibly on another forum. What was wrong with the replies that you got then? -- Kindest regards. Mark Lawrence. -- https://mail.python.org/mailman/listinfo/python-list
Plot map wit a white and black box
Hi, Please, I woudl like to plot a map like this figure. How can I do this using Python2.7 Thanks, Conrado -- https://mail.python.org/mailman/listinfo/python-list
Re: plot map wit box axes
> On Dec 23, 2017, at 3:27 PM, breamore...@gmail.com wrote: > > On Friday, December 22, 2017 at 3:42:58 PM UTC, jorge@cptec.inpe.br wrote: >> Hi, >> >> I use the PYTHON and IDL. In IDL I can plot a grid map like a this >> figure (mapa.png). Please, I would like know how can I plot my figure >> using PYTHON with the box around the figure. Like this that I plot using >> the IDL. >> >> Thanks > > Sorry but we can't see the image as it gets stripped off this text only > mailing list. What are you using to do the plot, matplotlib or smething > else? Can you show us the code you've used or your interactive session in > IDLE? > I’m 90% sure the OP really meant IDL, not IDLE. IDL (Interactive Data Language) is a long-time competitor to MatLab, and is widely used in various parts of the scientific community. (Don’t know if it is still true, but for years ALL the published images from the Hubble telescope had been processed through IDL.) Bill > -- > Kindest regards. > > Mark Lawrence. > -- > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: plot map wit box axes
On Friday, December 22, 2017 at 3:42:58 PM UTC, jorge@cptec.inpe.br wrote: > Hi, > > I use the PYTHON and IDL. In IDL I can plot a grid map like a this > figure (mapa.png). Please, I would like know how can I plot my figure > using PYTHON with the box around the figure. Like this that I plot using > the IDL. > > Thanks Sorry but we can't see the image as it gets stripped off this text only mailing list. What are you using to do the plot, matplotlib or smething else? Can you show us the code you've used or your interactive session in IDLE? -- Kindest regards. Mark Lawrence. -- https://mail.python.org/mailman/listinfo/python-list
plot map wit box axes
Hi, I use the PYTHON and IDL. In IDL I can plot a grid map like a this figure (mapa.png). Please, I would like know how can I plot my figure using PYTHON with the box around the figure. Like this that I plot using the IDL. Thanks -- https://mail.python.org/mailman/listinfo/python-list
Re: Performance of map vs starmap.
30.10.17 12:10, Kirill Balunov пише: Sometime ago I asked this question at SO [1], and among the responses received was paragraph: - `zip` re-uses the returned `tuple` if it has a reference count of 1 when the `__next__` call is made. - `map` build a new `tuple` that is passed to the mapped function every time a `__next__` call is made. Why can not `map` use the same approach as `zip`? Also it turns out that a faster solution looks not reasonable, since it requires additional calculations.. [1] https://stackoverflow.com/questions/46172018/perfomance- of-map-vs-starmap Sometime ago I had wrote a sample patch for using cached arg tuples in map() and several other functions [1]. It increased the speed in microbenchmarks up to 24-38%. But the code was too complex and fragile, I didn't want to commit it. Later, after numerous of attempts, this resulted in implementing by Victor Stinner the new private "fastcall" calling method which allowed to avoid creation a new tuple (and a dict for keyword arguments) in many cases. It was added just before releasing of 3.6 and was used in few places. In 3.7 it was optimized further. Now map() is faster than starmap()+zip(). Python 3.6: $ ./python -m timeit -s 'from operator import eq; from itertools import starmap; seq1 = [1]*1; seq2 = [1]*1' 'list(map(eq, seq1, seq2))' 1000 loops, best of 3: 559 usec per loop $ ./python -m timeit -s 'from operator import eq; from itertools import starmap; seq1 = [1]*1; seq2 = [1]*1' 'list(starmap(eq, zip(seq1, seq2)))' 1000 loops, best of 3: 399 usec per loop Python 3.7: $ ./python -m timeit -s 'from operator import eq; from itertools import starmap; seq1 = [1]*1; seq2 = [1]*1' 'list(map(eq, seq1, seq2))' 1000 loops, best of 5: 338 usec per loop $ ./python -m timeit -s 'from operator import eq; from itertools import starmap; seq1 = [1]*1; seq2 = [1]*1' 'list(starmap(eq, zip(seq1, seq2)))' 1000 loops, best of 5: 359 usec per loop [1] https://bugs.python.org/issue23507 -- https://mail.python.org/mailman/listinfo/python-list
Re: Performance of map vs starmap.
On Mon, 30 Oct 2017 09:10 pm, Kirill Balunov wrote: > Sometime ago I asked this question at SO [1], and among the responses > received was paragraph: > > - `zip` re-uses the returned `tuple` if it has a reference count of 1 when > the `__next__` call is made. > - `map` build a new `tuple` that is passed to the mapped function every > time a `__next__` call is made. > > Why can not `map` use the same approach as `zip`? It possibly could, if somebody could be bothered to make that micro-optimization. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Performance of map vs starmap.
Sometime ago I asked this question at SO [1], and among the responses received was paragraph: - `zip` re-uses the returned `tuple` if it has a reference count of 1 when the `__next__` call is made. - `map` build a new `tuple` that is passed to the mapped function every time a `__next__` call is made. Why can not `map` use the same approach as `zip`? Also it turns out that a faster solution looks not reasonable, since it requires additional calculations.. [1] https://stackoverflow.com/questions/46172018/perfomance- of-map-vs-starmap Thanks, - gdg -- https://mail.python.org/mailman/listinfo/python-list
Re: TypeError with map with no len()
On 9/25/2017 5:37 PM, Thomas Jollans wrote: On 25/09/17 18:44, john polo wrote: Python List, I am trying to make practice data for plotting purposes. I am using Python 3.6. The instructions I have are import matplotlib.pyplot as plt import math import numpy as np t = np.arange(0, 2.5, 0.1) y1 = map(math.sin, math.pi*t) If you use np.sin instead of math.sin, you don't have to use map: Most numpy functions operate elementwise on arrays (for example, you're multiplying math.pi with an array - something that wouldn't work with a list). Here's the numpy way of doing this: t = np.arange(0, 2.5, 0.1) y1 = np.sin(np.pi * t) Without using numpy at all, this might be t = [i * 0.1 for i in range(25)] y1 = [math.pi * a for a in t] The numpy way looks like a great alternative. Thank you, Thomas. John -- https://mail.python.org/mailman/listinfo/python-list
Re: TypeError with map with no len()
On 25/09/17 18:44, john polo wrote: > Python List, > > I am trying to make practice data for plotting purposes. I am using > Python 3.6. The instructions I have are > > import matplotlib.pyplot as plt > import math > import numpy as np > t = np.arange(0, 2.5, 0.1) > y1 = map(math.sin, math.pi*t) If you use np.sin instead of math.sin, you don't have to use map: Most numpy functions operate elementwise on arrays (for example, you're multiplying math.pi with an array - something that wouldn't work with a list). Here's the numpy way of doing this: t = np.arange(0, 2.5, 0.1) y1 = np.sin(np.pi * t) Without using numpy at all, this might be t = [i * 0.1 for i in range(25)] y1 = [math.pi * a for a in t] > plt.plot(t,y1) > > However, at this point, I get a TypeError that says > > object of type 'map' has no len() > > In [6]: t > Out[6]: > array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. , > 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2. , 2.1, > 2.2, 2.3, 2.4]) > In [7]: y1 > Out[7]: > In [8]: math.pi*t > Out[8]: > array([ 0., 0.31415927, 0.62831853, 0.9424778 , 1.25663706, > 1.57079633, 1.88495559, 2.19911486, 2.51327412, 2.82743339, > 3.14159265, 3.45575192, 3.76991118, 4.08407045, 4.39822972, > 4.71238898, 5.02654825, 5.34070751, 5.65486678, 5.96902604, > 6.28318531, 6.59734457, 6.91150384, 7.2256631 , 7.53982237]) > > At the start of creating y1, it appears there is an array to iterate > through for the math.sin function used in map(), but y1 doesn't appear > to be saving any values. I expected y1 to hold a math.sin() output for > each item in math.pi*t, but it appears to be empty. Am I > misunderstanding map()? Is there something else I should be doing > instead to populate y1? > > > John > -- https://mail.python.org/mailman/listinfo/python-list
Re: TypeError with map with no len()
On 9/25/2017 12:03 PM, Paul Moore wrote: You're using Python 3, and I suspect that you're working from instructions that assume Python 2. In Python 3, the result of map() is a generator, not a list (which is what Python 2's map returned). In order to get an actual list (which appears to be what you need for your plot call) you just need to call the list constructor: y1 = list(map(math.sin, math.pi*t)) Although given that you're using numpy, it may be that there's a more idiomatic numpy way of doing this. I'm not a numpy expert though, so I can't help on that. Paul Paul, Thank you very much for the explanation. best regards, John -- https://mail.python.org/mailman/listinfo/python-list
Re: TypeError with map with no len()
You're using Python 3, and I suspect that you're working from instructions that assume Python 2. In Python 3, the result of map() is a generator, not a list (which is what Python 2's map returned). In order to get an actual list (which appears to be what you need for your plot call) you just need to call the list constructor: y1 = list(map(math.sin, math.pi*t)) Although given that you're using numpy, it may be that there's a more idiomatic numpy way of doing this. I'm not a numpy expert though, so I can't help on that. Paul On 25 September 2017 at 17:44, john polo wrote: > Python List, > > I am trying to make practice data for plotting purposes. I am using Python > 3.6. The instructions I have are > > import matplotlib.pyplot as plt > import math > import numpy as np > t = np.arange(0, 2.5, 0.1) > y1 = map(math.sin, math.pi*t) > plt.plot(t,y1) > > However, at this point, I get a TypeError that says > > object of type 'map' has no len() > > In [6]: t > Out[6]: > array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. , > 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2. , 2.1, > 2.2, 2.3, 2.4]) > In [7]: y1 > Out[7]: > In [8]: math.pi*t > Out[8]: > array([ 0., 0.31415927, 0.62831853, 0.9424778 , 1.25663706, > 1.57079633, 1.88495559, 2.19911486, 2.51327412, 2.82743339, > 3.14159265, 3.45575192, 3.76991118, 4.08407045, 4.39822972, > 4.71238898, 5.02654825, 5.34070751, 5.65486678, 5.96902604, > 6.28318531, 6.59734457, 6.91150384, 7.2256631 , 7.53982237]) > > At the start of creating y1, it appears there is an array to iterate through > for the math.sin function used in map(), but y1 doesn't appear to be saving > any values. I expected y1 to hold a math.sin() output for each item in > math.pi*t, but it appears to be empty. Am I misunderstanding map()? Is there > something else I should be doing instead to populate y1? > > > John > > -- > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: TypeError with map with no len()
On 2017-09-25, john polo wrote: > Python List, > > I am trying to make practice data for plotting purposes. I am using > Python 3.6. The instructions I have are > > import matplotlib.pyplot as plt > import math > import numpy as np > t = np.arange(0, 2.5, 0.1) > y1 = map(math.sin, math.pi*t) > plt.plot(t,y1) > > However, at this point, I get a TypeError that says > > object of type 'map' has no len() you probably need to convert y1 from a 'map' object (a type of an iterator) to an array. y1 = map(math.sin, math.pi*t) y1 = np.fromiter(y1, float) -- Grant Edwards grant.b.edwardsYow! Will this never-ending at series of PLEASURABLE gmail.comEVENTS never cease? -- https://mail.python.org/mailman/listinfo/python-list
Re: TypeError with map with no len()
john polo wrote:
> Python List,
>
> I am trying to make practice data for plotting purposes. I am using
> Python 3.6. The instructions I have are
>
> import matplotlib.pyplot as plt
> import math
> import numpy as np
> t = np.arange(0, 2.5, 0.1)
> y1 = map(math.sin, math.pi*t)
> plt.plot(t,y1)
>
> However, at this point, I get a TypeError that says
>
> object of type 'map' has no len()
>
> In [6]: t
> Out[6]:
> array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ,
> 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2. , 2.1,
> 2.2, 2.3, 2.4])
> In [7]: y1
> Out[7]:
> In [8]: math.pi*t
> Out[8]:
> array([ 0., 0.31415927, 0.62831853, 0.9424778 , 1.25663706,
> 1.57079633, 1.88495559, 2.19911486, 2.51327412, 2.82743339,
> 3.14159265, 3.45575192, 3.76991118, 4.08407045, 4.39822972,
> 4.71238898, 5.02654825, 5.34070751, 5.65486678, 5.96902604,
> 6.28318531, 6.59734457, 6.91150384, 7.2256631 , 7.53982237])
>
> At the start of creating y1, it appears there is an array to iterate
> through for the math.sin function used in map(), but y1 doesn't appear
> to be saving any values. I expected y1 to hold a math.sin() output for
> each item in math.pi*t, but it appears to be empty. Am I
> misunderstanding map()? Is there something else I should be doing
> instead to populate y1?
map() is "lazy" in Python 3, i. e. it calculates the values when you iterate
over it, not before:
>>> def noisy_square(x):
... print("calculating {0} * {0}".format(x))
... return x * x
...
>>> squares = map(noisy_square, [1, 3, 2])
>>> for y in squares:
... print(y)
...
calculating 1 * 1
1
calculating 3 * 3
9
calculating 2 * 2
4
While you can convert y1 to a list with
y1 = list(y1)
the better approach is to use numpy:
y1 = np.sin(t * math.pi)
Now y1 is a numpy.array and can be fed to pyplot.plot() directly.
--
https://mail.python.org/mailman/listinfo/python-list
Re: TypeError with map with no len()
On 9/25/17 12:44 PM, john polo wrote: Python List, I am trying to make practice data for plotting purposes. I am using Python 3.6. The instructions I have are import matplotlib.pyplot as plt import math import numpy as np t = np.arange(0, 2.5, 0.1) y1 = map(math.sin, math.pi*t) plt.plot(t,y1) However, at this point, I get a TypeError that says object of type 'map' has no len() In [6]: t Out[6]: array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. , 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2. , 2.1, 2.2, 2.3, 2.4]) In [7]: y1 Out[7]: In [8]: math.pi*t Out[8]: array([ 0. , 0.31415927, 0.62831853, 0.9424778 , 1.25663706, 1.57079633, 1.88495559, 2.19911486, 2.51327412, 2.82743339, 3.14159265, 3.45575192, 3.76991118, 4.08407045, 4.39822972, 4.71238898, 5.02654825, 5.34070751, 5.65486678, 5.96902604, 6.28318531, 6.59734457, 6.91150384, 7.2256631 , 7.53982237]) At the start of creating y1, it appears there is an array to iterate through for the math.sin function used in map(), but y1 doesn't appear to be saving any values. I expected y1 to hold a math.sin() output for each item in math.pi*t, but it appears to be empty. Am I misunderstanding map()? Is there something else I should be doing instead to populate y1? In Python 2, map() produced a list. In Python 3, it returns a lazy iterator instead. You can run the iterator with list(): y1 = list(map(math.sin, math.pi*t)) --Ned. -- https://mail.python.org/mailman/listinfo/python-list
Re: TypeError with map with no len()
On 9/25/2017 12:44 PM, john polo wrote: Python List, I am trying to make practice data for plotting purposes. I am using Python 3.6. The instructions I have are import matplotlib.pyplot as plt import math import numpy as np t = np.arange(0, 2.5, 0.1) y1 = map(math.sin, math.pi*t) Change to y1 = list(map(math.sin, math.pi*t)) or y1 = tuple(map(math.sin, math.pi*t)) (Does not make much difference.) plt.plot(t,y1) or plt.plot(t, list(y1)) However, at this point, I get a TypeError that says object of type 'map' has no len() In 3.x, map returns an iterator. plot needs a sequence. In [6]: t Out[6]: array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. , 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2. , 2.1, 2.2, 2.3, 2.4]) In [7]: y1 Out[7]: In [8]: math.pi*t Out[8]: array([ 0. , 0.31415927, 0.62831853, 0.9424778 , 1.25663706, 1.57079633, 1.88495559, 2.19911486, 2.51327412, 2.82743339, 3.14159265, 3.45575192, 3.76991118, 4.08407045, 4.39822972, 4.71238898, 5.02654825, 5.34070751, 5.65486678, 5.96902604, 6.28318531, 6.59734457, 6.91150384, 7.2256631 , 7.53982237]) At the start of creating y1, it appears there is an array to iterate through for the math.sin function used in map(), but y1 doesn't appear to be saving any values. I expected y1 to hold a math.sin() output for each item in math.pi*t, but it appears to be empty. Am I misunderstanding map()? See comment 3 above. Is there something else I should be doing instead to populate y1? See comment 1 (or 2) above. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
TypeError with map with no len()
Python List, I am trying to make practice data for plotting purposes. I am using Python 3.6. The instructions I have are import matplotlib.pyplot as plt import math import numpy as np t = np.arange(0, 2.5, 0.1) y1 = map(math.sin, math.pi*t) plt.plot(t,y1) However, at this point, I get a TypeError that says object of type 'map' has no len() In [6]: t Out[6]: array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. , 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2. , 2.1, 2.2, 2.3, 2.4]) In [7]: y1 Out[7]: In [8]: math.pi*t Out[8]: array([ 0. , 0.31415927, 0.62831853, 0.9424778 , 1.25663706, 1.57079633, 1.88495559, 2.19911486, 2.51327412, 2.82743339, 3.14159265, 3.45575192, 3.76991118, 4.08407045, 4.39822972, 4.71238898, 5.02654825, 5.34070751, 5.65486678, 5.96902604, 6.28318531, 6.59734457, 6.91150384, 7.2256631 , 7.53982237]) At the start of creating y1, it appears there is an array to iterate through for the math.sin function used in map(), but y1 doesn't appear to be saving any values. I expected y1 to hold a math.sin() output for each item in math.pi*t, but it appears to be empty. Am I misunderstanding map()? Is there something else I should be doing instead to populate y1? John -- https://mail.python.org/mailman/listinfo/python-list
Re: Cross-language comparison: function map and similar
On Thu, 17 Aug 2017 12:53 am, Steve D'Aprano wrote: [...] > Are there language implementations which evaluate the result of map() (or its > equivalent) in some order other than the obvious left-to-right first-to-last > sequential order? Is that order guaranteed by the language, or is it an > implementation detail? Thanks to everyone who answered. It also confirmed my suspicion: apart from explicitly parallel languages, and specialised parallel versions of map, mainstream programming languages do not treat the execution order of map() as an implementation detail. At least not the languages which people here (reading this thread, and motivated to reply) are familiar with. I take this as moderate evidence that programming languages do not *actually* treat execution order of map and equivalent as an implementation detail, even if they reserve the right to. Instead, they generally treat it as semantically a sequential loop[1]. Apart from explicitly parallel languages, if you want a parallel or concurrent map, you need to either call a specialised version, or specify an execution policy to set the execution model, or equivalent. What you can't do, in any language mentioned here, is simply call the regular, standard map function. Thanks again to everyone who answered, and if anyone want to point out an exception, its not too late :-) [1] Whether implemented as a loop, or by recursion, or something else. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Cross-language comparison: function map and similar
Op 2017-08-16, Steve D'Aprano schreef : > Are there language implementations which evaluate the result of map() > (or its equivalent) in some order other than the obvious left-to-right > first-to-last sequential order? Is that order guaranteed by the > language, or is it an implementation detail? > > Standard library functions implementing an explicitly "parallel map" > or "threaded map" are also relevant. (Less interested in third-party > libraries, unless they're practically a standard for the language in > question.) C++17 has such a facility in its standard library. std::transform (and many other functions operating on sequences, but you asked for a map() equivalent) takes an optional "execution_policy" parameter which indicates if the operation should be run sequentially (the default) or can be parallellized. See: http://en.cppreference.com/w/cpp/algorithm/transform Stephan -- https://mail.python.org/mailman/listinfo/python-list
Re: Cross-language comparison: function map and similar
On Thu, 17 Aug 2017 03:54 pm, Pavol Lisy wrote: > Is it guaranteed in python? Or future version could implement map with > something like subscriptability "propagation"? > >>>>range(1_000_000_000_000_000_000)[-1] > > >>>> map(lambda a:a+1,range(1_000_000_000_000_000_000))[-1] > TypeError: 'map' object is not subscriptable I don't recognise the term "subscriptability propagation". Given a suitable deprecation period, or another backwards-incompatible release like Python 3, anything is possible. But some things are very unlikely, such as another backwards-incompatible release, or map supporting indexing. Python's map() now returns an iterator, which means it must yield items one by one, in order. Because it is an iterator, in general it cannot know what the next item will be until it evaluates it. Unlike range, it cannot jump ahead In principle we could have map() return a magic iterator that delegated indexing to the underlying sequence, but that's not part of the iterator API and the benefit seems small. So, possible but unlikely. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Cross-language comparison: function map and similar
On Thu, Aug 17, 2017 at 3:54 PM, Pavol Lisy wrote: > On 8/16/17, Steve D'Aprano wrote: >> Over in another thread, we've been talking about comprehensions and their >> similarities and differences from the functional map() operation. >> >> Reminder: >> >> map(chr, [65, 66, 67, 68]) >> >> will return ['A', 'B', 'C']. >> >> My questions for those who know languages apart from Python: >> >> Are there language implementations which evaluate the result of map() (or >> its >> equivalent) in some order other than the obvious left-to-right first-to-last >> sequential order? Is that order guaranteed by the language, or is it an >> implementation detail? > > Is it guaranteed in python? Or future version could implement map with > something like subscriptability "propagation"? > >>>>range(1_000_000_000_000_000_000)[-1] > 9999 > >>>> map(lambda a:a+1,range(1_000_000_000_000_000_000))[-1] > TypeError: 'map' object is not subscriptable I think it is, partly because of the behaviour of map with additional arguments. If you want something subscriptable, you probably don't want an iterable, but a mapping type (it's no coincidence that the words are similar). The Python map() function expects a function and one or more iterables, and returns an iterator; but if you instead give it a function and a (subscriptable) collections, you could have it be a collection. (I'm not sure what to do about multiple collections. I guess you'd do something similar to map() - if any subcollection raises, Map raises. Not implemented here though.) class Map(dict): def __init__(self, func, coll): self.func, self.coll = func, coll def __missing__(self, key): self[key] = self.func(self.coll[key]) return self[key] This has the same subscripting semantics as the underlying collection, and will cache any result it sees. But watch out! It can't know about subscript aliasing, and will treat a key of -1 as completely different from any positive subscript. It also doesn't iterate the same way as the underlying collection does: >>> spam = Map(print, range(10)) >>> list(spam) [] Without knowing exactly what kind of "thing" you're mapping over, it's impossible to get everything right. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Cross-language comparison: function map and similar
On 8/16/17, Steve D'Aprano wrote: > Over in another thread, we've been talking about comprehensions and their > similarities and differences from the functional map() operation. > > Reminder: > > map(chr, [65, 66, 67, 68]) > > will return ['A', 'B', 'C']. > > My questions for those who know languages apart from Python: > > Are there language implementations which evaluate the result of map() (or > its > equivalent) in some order other than the obvious left-to-right first-to-last > sequential order? Is that order guaranteed by the language, or is it an > implementation detail? Is it guaranteed in python? Or future version could implement map with something like subscriptability "propagation"? >>>range(1_000_000_000_000_000_000)[-1] >>> map(lambda a:a+1,range(1_000_000_000_000_000_000))[-1] TypeError: 'map' object is not subscriptable -- https://mail.python.org/mailman/listinfo/python-list
Re: Cross-language comparison: function map and similar
Steve D'Aprano writes: > Are there language implementations which evaluate the result of map() > (or its equivalent) in some order other than the obvious left-to-right > first-to-last sequential order? Is that order guaranteed by the > language, or is it an implementation detail? Haskell just gives back an unevaluated thunk. The elements are evaluated in whatever order you happen to use them in. Since the evaluation isn't supposed to have observable side effects, there's no way to tell the order. There are also parallel versions (Control.Parallel.Strategies.parMap etc.) and an extension that recognizes "racing stripes" on the square brackets to make list comprehensions run faster: foo = [| sqrt(x) | x <- [1.0 .. 1000.0] |] parallelizes the calculation and offloads it to a GPU or other vector processor. You might also like this old post https://donsbot.wordpress.com/2007/11/29/use-those-extra-cores-and-beat-c-today-parallel-haskell-redux/ "map" is conceptually the lifting of a function from a given type, to the type under the action of some functor. For historical reasons map only works on lists while fmap works on arbitrary functors, but they are in principle the same thing. Depending on what the functor does, the whole concept of left-to-right order might be meaningless. -- https://mail.python.org/mailman/listinfo/python-list
Re: Cross-language comparison: function map and similar
On 8/16/2017 10:53 AM, Steve D'Aprano wrote: Over in another thread, we've been talking about comprehensions and their similarities and differences from the functional map() operation. Reminder: map(chr, [65, 66, 67, 68]) will return ['A', 'B', 'C']. The comprehension 'while' proposal is for future Python 3. Asking your while question the restricted context of Python 2 encouraged the restricted list-based or concrete-collection-based thinking about comprehensions that you got in some of the answers. The above snipper is only true for Python-2 . In Python 3, >>> oracle = map(chr, [65, 66, 67, 68]) >>> oracle >>> next(oracle) 'A' >>> next(oracle) 'B' >>> next(oracle) 'C' When oracle is asked for the next value, it asks base_oracle = iter(iterable) for a value. If base_oracle gives one, oracle gives func(value). If base_iterable raises StopIteration, so does oracle. This process is inherently sequential. It remains so with multiple input iterables. >>> next(oracle) 'D' >>> next(oracle) Traceback (most recent call last): File "", line 1, in next(oracle) StopIteration Even in pre-2.2 Python, map's input was not restricted to being a list, but could be any 'sequence'. From 2.0 doc: "The list arguments may be *any kind of sequence*" [emphasis added]. I am rather sure that in this context, a 'sequence' was merely something that could be indexed and that a pre-defined length was not required. This means that map could use the old iterator protocol. So 'sequence' meant 'iterator' in current terms. The old spelling of 'next(iterator)' was 'iterator[i]' where i is 0 for the first request for an object and incremented thereafter. The 'sequence' was free to ignore i when generating the return object. It could also use external input. My questions for those who know languages apart from Python: Are there language implementations which evaluate the result of map() (or its equivalent) in some order other than the obvious left-to-right first-to-last sequential order? If map's collection input is unbounded, as with Python, a right-to-left or completely parallel order is not possible. If the input oracle creates objects on demand, as Python allows, then doing anything other that applying func as objects are made available requires internal storage. Asking an oracle for millions of objects when only the first few are needed, is foolish. Storing millions or billions of objects all at once when there are only needed one at a time is also foolish. Which is why map and filter were changed in Python 3. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Cross-language comparison: function map and similar
On Wednesday, August 16, 2017 at 8:24:46 PM UTC+5:30, Steve D'Aprano wrote: > Over in another thread, we've been talking about comprehensions and their > similarities and differences from the functional map() operation. > > Reminder: > > map(chr, [65, 66, 67, 68]) > > will return ['A', 'B', 'C']. > > My questions for those who know languages apart from Python: > > Are there language implementations which evaluate the result of map() (or its > equivalent) in some order other than the obvious left-to-right first-to-last > sequential order? Is that order guaranteed by the language, or is it an > implementation detail? There are dozens of parallel/concurrent languages: https://en.wikipedia.org/wiki/List_of_concurrent_and_parallel_programming_languages > > Standard library functions implementing an explicitly "parallel map" > or "threaded map" are also relevant. Here's the peach (parallel-each) 'adverb in Q/K http://code.kx.com/wiki/Reference/peach In more mainstream languages: parallel map in Julia https://docs.julialang.org/en/latest/manual/parallel-computing Haskell's parmap et al: http://chimera.labs.oreilly.com/books/123000929/ch03.html#sec_par-kmeans-perf -- https://mail.python.org/mailman/listinfo/python-list
Re: Cross-language comparison: function map and similar
On Thu, 17 Aug 2017 12:53 am, Steve D'Aprano wrote: > map(chr, [65, 66, 67, 68]) > > will return ['A', 'B', 'C']. Of course I meant ['A', 'B', 'C', 'D']. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Cross-language comparison: function map and similar
Over in another thread, we've been talking about comprehensions and their similarities and differences from the functional map() operation. Reminder: map(chr, [65, 66, 67, 68]) will return ['A', 'B', 'C']. My questions for those who know languages apart from Python: Are there language implementations which evaluate the result of map() (or its equivalent) in some order other than the obvious left-to-right first-to-last sequential order? Is that order guaranteed by the language, or is it an implementation detail? Standard library functions implementing an explicitly "parallel map" or "threaded map" are also relevant. (Less interested in third-party libraries, unless they're practically a standard for the language in question.) E.g. given the map above, Blub language might promise to evaluate chr(65), chr(66), chr(67) and chr(68) in parallel if your computer has four cores. Or some Blub language implementation may unroll the above map to the equivalent of this pseudo-code: array = [None, None, None, None] array[3] = chr(68) array[2] = chr(67) array[1] = chr(66) array[0] = chr(65) I'm not terribly interested in hypothetical "Blub language doesn't specify evaluation order, so some implementation could do this". I'm more interested in actual concrete examples of mapping implementations which don't operate in the obvious first-to-last order. For example, Fortran has "do concurrent": do concurrent(i = 1:n) a(i) = a(i+m)+b(i) end do which tells the compiler that it can run the iterations in any order, or parallelize them and run them in parallel across threads, etc: https://www.hpcwire.com/2015/04/06/compilers-and-more-the-past-present-and-future-of-parallel-loops/ Wikipedia mentions a few specialised parallel processing languages which have equivalent forms of parallel map: https://en.wikipedia.org/wiki/Map_%28parallel_pattern%29 Wolfram Language (Mathematica?) has a parallel map: http://reference.wolfram.com/language/ref/ParallelMap.html as does Clojure: https://clojuredocs.org/clojure.core/pmap Any others? I'm especially interested in cases of languages where their regular map() function is performed in parallel, or out of order, *without* the performance hit that the Clojure docs warn about: "Only useful for computationally intensive functions where the time of f dominates the coordination overhead." -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Is "two different input map to one unique target called nonlinear case " or "one unique input map to two different target called nonlinear case"? Which learning method can train these two cases or no
Is "two different input map to one unique target called nonlinear case " or "one unique input map to two different target called nonlinear case"? Which learning method can train these two cases or no method to train one of case? -- https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
On Sun, Nov 6, 2016 at 12:50 AM, Arthur Havlicek wrote: > 2016-11-05 12:47 GMT+01:00 Chris Angelico : > >> On Sat, Nov 5, 2016 at 9:50 PM, Arthur Havlicek >> >> But here's the thing. For everyone who writes a decorator function, >> there could be dozens who use it. >> > > The day that one guy leaves the team, suddenly you have code that's become > a bit tricky to maintain. True, and this can become a problem when those dozens have no comprehension of how these features work. But fortunately, all it takes is for one person to step up and learn how decorators are written, and the problem is solved. (And it's not that hard. We teach decorator authorship in our Web Programming In Python course. Not in a huge amount of detail, but enough that a student will be able to carefully tease apart the code of a decorator function and figure out what it's actually doing.) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
2016-11-05 12:47 GMT+01:00 Chris Angelico : > On Sat, Nov 5, 2016 at 9:50 PM, Arthur Havlicek > > But here's the thing. For everyone who writes a decorator function, > there could be dozens who use it. > The day that one guy leaves the team, suddenly you have code that's become a bit tricky to maintain. Our philosophy is that everyone should be able to contribute anywhere although I agree with you: there is such thing as using a decorator as a black box, and even diving into it isn't that hard. Notice I didn't say we don't use these features at all, we just tend to avoid them when a more familiar approach exist. Even so, maybe we are wrong to do so, and the code would be clearer if we used more features. Maybe we just lack the expertise and we are indeed stuck in an old mindset, after all. Anyway, I have some directives to make my code easy to access and must compose with what I would call industrial constraints. As a gambler I can't refuse that bet proposal, no sir :) Expect me to come back with a piece of code and/or questions about CPython engine. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
On Sat, Nov 5, 2016 at 9:50 PM, Arthur Havlicek
wrote:
> Pick 10 programmers for hire and count how many know how to write a
> decorator. If you have specified you needed python specialists, you may
> have 3-4. If not, you are lucky to find even one.
By "write a decorator", I presume you mean implement the decorator
function, because anyone who's used Flask will have used "@app.route",
for instance.
But here's the thing. For everyone who writes a decorator function,
there could be dozens who use it. How many people *on this mailing
list* know how to implement namedtuple, vs how how many know how to
use it? How many know how to properly implement a cache, vs how many
can slap "@lru_cache()" on top of a function? For the most utterly
extreme example possible, how many people in the world know how to
encrypt data securely, vs how many know how to put "https:" into their
browsers? When you look at "programmers for hire", you get a small
number of brilliant experts with huge amounts of experience, and the
rest are split between mid-level people looking for a new job, and
entry-level people straight out of college. (Depending on your
environment, the stats could easily be skewed either direction.) Most
entry-level programmers are not going to have experience with writing
decorator functions - but they don't need it. (Not that it's that
hard. They're functions that accept and return functions, that's all.)
> This is easy but time-consuming, I could roll my implementation and
> showcase a few benchs. I am very confident that is the case. I would have
> bet that I would need to do it at some point, but I wanted to poll opinions
> a bit beforehand.
>
> About your bench: I don't really know why you are surprised the for loop is
> slower. That's the result of comprehension being native while for loop
> isn't. That does not mean writing to a copy would save time for exchange of
> memory. In my opinion, the fact that we will win something is guaranteed
> because we save a copy call and do the exact same operation. There is
> nothing like cache magic optimization that could happen because the mapping
> needs to read the first list anyway. Nor we need a temporary buffer to
> cache operations since they are independent. Really, I am ready to make a
> serious bet.
Okay! *Do* make that bet. Gamble the most precious currency there is:
developer time. How much are you prepared to bet? Two hours? Ten
hours? Spend that much time writing the implementation and
benchmarking it. If you're confident that this really is such a win,
the time won't be lost - you'll end up with a viable patch for
inclusion. If you lose the bet, well, that's what betting is like.
Sometimes you lose.
I may sound cynical and critical from the above ("show me"), but in
reality, I would love to see the results of your benchmark. Improving
performance is a good thing. I just want to know that it's actually
going to happen.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
2016-11-05 9:42 GMT+01:00 Steve D'Aprano : > > I don't know who you are quoting there. It is considered impolite to quote > people without giving attribution, and makes it harder to respond. > My bad. I was unaware of that. This was quoted from Ned Batchelder's mali. 2016-11-05 9:42 GMT+01:00 Steve D'Aprano : > I trust that isn't actually the case, but by describing decorators and > lambda as too fancy to use, you're giving a good impression of somebody who > doesn't know what they're doing. > I was a bit humorous here, making fun about the fact I do not write anything sophisticated. However, the fact that we do avoid them is entirely true ! We do so because these concepts are foreign to the average developer (lambda being a bit apart: it's easily understood, but functional constructs may not, so its a result of us avoiding functional constructs as well). Pick 10 programmers for hire and count how many know how to write a decorator. If you have specified you needed python specialists, you may have 3-4. If not, you are lucky to find even one. And where I live we don't have the luxury of finding competent Python experts by kicking a tree. In our open space, we are 6 devs (+ one ex-dev being my manager), only 2 had previous Python experience: me and the CEO. And the other 4 are doing fine ! Locking ourselves in Python-specific semantics, over time, will make us loose precious dev time. The best code is the one everyone can understand. That is why I point this as "complex, non-obvious", and my manager wouldn't have so kind words (but he is a Java fan, so he doesn't count.) 2016-11-05 9:42 GMT+01:00 Steve D'Aprano : > - you would have to demonstrate a good reason to think that this new > feature > will actually be more efficient and faster > This is easy but time-consuming, I could roll my implementation and showcase a few benchs. I am very confident that is the case. I would have bet that I would need to do it at some point, but I wanted to poll opinions a bit beforehand. 2016-11-05 9:42 GMT+01:00 Steve D'Aprano : > In other words, in my opinion: > > - you would need to prove that there is a widespread need for in-place > modification of lists, where the lists are big enough that using a > temporary list is not practical > However this is tricky, I may very well be an exception. About your bench: I don't really know why you are surprised the for loop is slower. That's the result of comprehension being native while for loop isn't. That does not mean writing to a copy would save time for exchange of memory. In my opinion, the fact that we will win something is guaranteed because we save a copy call and do the exact same operation. There is nothing like cache magic optimization that could happen because the mapping needs to read the first list anyway. Nor we need a temporary buffer to cache operations since they are independent. Really, I am ready to make a serious bet. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
I've been giving your proposal a bit more thought, and while I can't say I'm
really keep on the idea, I have warmed slightly to it.
On Fri, 4 Nov 2016 07:29 am, Arthur Havlicek wrote:
> I understand that, the cost of change is such that it's very unlikely
> something like this ever goes into Python, but I feel like the interest of
> the proposition is being underestimated here, that's why I'm going to
> argue a few points and give a bit more context as needed.
>
>> While mapping and filtering are common operations, I'm not sure mapping
>> and filtering and then reassigning back to the original sequence is
>> especially common.
>
> It depends of your context. On the last 3 months, I stumbled across this
> at least 3 times,
I don't know who you are quoting there. It is considered impolite to quote
people without giving attribution, and makes it harder to respond. But for
what it's worth, I agree with this person.
In my code, it is quite common for me to map back to the same *variable*,
for example I might write something like this:
def process(alist):
alist = [float(x) for x in alist]
...
But that is not the same as *in-place* modification of the list. I would
very rarely modify the list in place, but if I did, I would write it like
this:
alist[:] = [float(x) for x in alist]
In my experience, wanting to modify a list in-place without creating a
temporary version first is unusual. And *needing* to do so (rather than
doing so as premature optimization) is even rarer.
That's my experience. But if you can demonstrate the need for in-place
modifications, that changes the situation somewhat.
> which is 3 times more than I used a lambda or a
> metaclass or a decorator or other fancy language feature that we simply
> avoid whenever possible.
You aren't helping your case by describing "lambda or ... decorator" as
fancy language features to be avoided.
The impression that gives (hopefully this is the wrong impression!) is that
you are a barely competent Python developer, fearful and suspicious of some
rather simple, powerful and widely-used features, stuck in an ancient 1980s
programming paradigm.
I trust that isn't actually the case, but by describing decorators and
lambda as too fancy to use, you're giving a good impression of somebody who
doesn't know what they're doing.
[...]
> The reason I'm being especially impacted by this is because I am
> maintainer of a decent-sized Python application (~50-100K lines of code)
> that extensively uses lists and dictionaries. We value "low level" data
> manipulation and efficiency a lot more than complex, non-obvious
> constructs.
And what do you consider "complex, non-obvious"?
[...]
> Like most Python programmers, I'm not in the case of needing a performance
> boost very bad, but that does not mean I disregard performance entirely.
> The need of performance is not so binary that it either don't matter at
> all or is enough to motivate a rewrite.
Indeed.
But you're arguing for a new feature on the basis that it will boost
performance, without giving any reason to think that it actually will lead
to a performance improvement!
I know that there is a chicken-and-egg problem here. Nobody wants to spend
time writing a new feature if there's no possibly chance for it to be
accepted, but when your sole argument for a new feature is that it will be
more efficient (in both memory and time!), then you need to prove that is
the case!
In other words, in my opinion:
- you would need to prove that there is a widespread need for in-place
modification of lists, where the lists are big enough that using a
temporary list is not practical;
- you would have to demonstrate a good reason to think that this new feature
will actually be more efficient and faster
before this feature would be seriously considered for addition to the
language.
You might think that it is obvious that in-place modification MUST be faster
since it avoids making a temporary copy, but that's not obviously true at
all! Not for a high-level language like Python. Its often the case that
algorithms can exchange space for time (use more memory to save time, or
use more time to save memory). It may be that this is one of the times.
Even when doing (nearly) everything in pure Python, making a new list and
then doing a slice assignment is virtually as fast as writing directly to
the original list:
py> from timeit import Timer
py> setup = "data = list(range(1))"
py> t1 = Timer("""for i, a in enumerate(data):
... data[i] = a+1
... """, setup=setup)
py>
py> t2 = Timer("""tmp = [None]*len(data)
... for i, a in enumerate(data):
... tmp[i] = a+1
... data[:] = tmp
... """, setup=setup)
py
Re: Pre-pep discussion material: in-place equivalents to map and filter
> I don't think that justifies the claim of "especially > bad", which to me implies something much worse. Quicksort has built its popularity by performing better by "a mere factor two" better than mergesort and heapsort. It became the reference sorting algorithm even though its worst case complexity is worse than its competitors. You can insurge about the fact I call a factor two to be especially bad, and you are right it is an hyperbole in the general case, but I am still right in the specific context of a frequently used linear algorithm. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
On Sat, Nov 5, 2016 at 11:42 AM, Steve D'Aprano wrote: > On Fri, 4 Nov 2016 08:34 am, Chris Angelico wrote: > > [...] >> List comps themselves involve one function call (zero in Py2). What >> you do inside the expression is your business. Do you agree that list >> comps don't have the overhead of opening and closing files? > > /tongue firmly in cheek > > I'd like to see you run a Python script containing a list comprehension > without opening and closing the .py file. > > :-P You got me! Let's call it a night. -- Kristoff (nearly my namesake) > Okay, I see what you are getting at now: in CPython 3, list comprehensions > are implemented in such a way that the list comprehension requires a > minimum of one function call, while list(map(func, iterable))) requires a > minimum of O(N)+2 function calls: a call to list, a call to map, and a call > to func for each of N values. > > That's *technically* true, but it is an implementation detail. I'm sure that > some day PyPy or Nuitka or maybe even CPython itself will start in-lining > at least some functions (if PyPy doesn't already do so). That would be an > obvious optimization to apply to map() and filter(). Mmmm, interesting. The fact still remains that map depends on some kind of "object representation" of a block of code (since it's being passed to some other function), where a comprehension can be implemented as an actual expression. So either Python-the-language needs a way to pass around lightweight blocks of code, or the interpreter (for some instance of 'the interpreter') needs to recognize the function calls and optimize them away, or list comps will always have an inherent advantage over map. > As far as the speed of map() versus list comps, my micro-benchmarks show > that at least on my computer, map() can be marginally but consistently > faster than a list comp once you equalise that cost of function calls, that > is, if the list comp explicitly calls a function. I don't think that speed > difference is significant, so let's just call them "equally fast" when > comparing similar cases: > > [func(obj) for obj in iterable] > > map(func, iterable) Right, which is why a lot of style guides recommend against the first form, *in this specific instance*. Using map with a lambda function, or a comprehension with nothing but a function call, is rightly called out in code review. > I didn't read the OP as making a specific claim about these two *specific* > map and filter examples: > > lst = map (lambda x: x*5, lst) > lst = filter (lambda x: x%3 == 1, lst) > > I read these as mere examples of a general claim that map and > filter "perform especially bad in CPython compared to a comprehension". > ... I don't think that justifies the claim of "especially > bad", which to me implies something much worse. If you're going to describe > a mere factor of two as "especially bad", what words do we have left for > something that is ten thousand times slower? > > As the wisest man in the universe once said, hyperbole is the most terrible, > awful crime against humanity. > > *wink* Ah, now we get to the point where we disagreed. I was responding to a misunderstanding of your position - I thought you disagreed that the performance difference could even be significant, but you were arguing against the "especially bad". Gotcha. In that case, I believe we're in agreement; even a two-to-one difference isn't "especially bad" here, and that would be an extreme case. >> But this conclusion I agree with. There is a performance difference, >> but it is not overly significant. Compared to the *actual work* >> involved in the task (going through one list and doing some operation >> on each operation), the difference between map and a comprehension is >> generally going to be negligible. > > I wouldn't go quite so far as to say "negligible" -- a factor of two speed > up on a large list is not something to be sneezed at. But I think we're > converging on agreement: list comps and map/filter typically have > comparable performance, and as the cost of the work done increases, the > extra overhead of a function call becomes less and less significant. I said negligible because the factor of two disappears almost completely when you add a large constant factor to it. What's the performance of these? def lie(n): """it's not a fib, honest""" if n < 2: return 3 return lie(n-1) + lie(n-2) mapped = list(map(lie, range(50))) comprehended = [lie(x) for x in range(50)] badly_mapped = list(map(lambda x: lie(x), range(50))) By any reasonable
Re: Pre-pep discussion material: in-place equivalents to map and filter
On Fri, 4 Nov 2016 08:34 am, Chris Angelico wrote:
[...]
> List comps themselves involve one function call (zero in Py2). What
> you do inside the expression is your business. Do you agree that list
> comps don't have the overhead of opening and closing files?
/tongue firmly in cheek
I'd like to see you run a Python script containing a list comprehension
without opening and closing the .py file.
:-P
Okay, I see what you are getting at now: in CPython 3, list comprehensions
are implemented in such a way that the list comprehension requires a
minimum of one function call, while list(map(func, iterable))) requires a
minimum of O(N)+2 function calls: a call to list, a call to map, and a call
to func for each of N values.
That's *technically* true, but it is an implementation detail. I'm sure that
some day PyPy or Nuitka or maybe even CPython itself will start in-lining
at least some functions (if PyPy doesn't already do so). That would be an
obvious optimization to apply to map() and filter().
As far as the speed of map() versus list comps, my micro-benchmarks show
that at least on my computer, map() can be marginally but consistently
faster than a list comp once you equalise that cost of function calls, that
is, if the list comp explicitly calls a function. I don't think that speed
difference is significant, so let's just call them "equally fast" when
comparing similar cases:
[func(obj) for obj in iterable]
map(func, iterable)
But of course you're right that list comprehensions give you the opportunity
to avoid that function call -- at least in CPython. I already agreed with
that, but to emphasise what I've already agreed, I'll say it again :-)
If you can manually in-line func() as a single expression inside the list
comprehension:
[(spam + len(obj.eggs))*2 for obj in iterable]
then you can expect to save the cost of N function calls, which may be
significant. As I said earlier, that's why we have list comps.
(But on the other hand, if the expression is expensive enough, the cost of
an extra function call may be utterly insignificant.)
The point that I am trying to make is that none of these facts justifies the
claim that map() performs "especially bad" compared to list comprehensions.
According to my tests, *at worst* map() will be a bit better than half as
fast as a list comprehension, and at best just as fast if not slightly
faster.
>> Here's some timing results using 3.5 on my computer. For simplicity, so
>> folks can replicate the test themselves, here's the timing code:
>>
>>
>> from timeit import Timer
>> setup = """data = list(range(1))
>> def func(x): # simulate some calculation
>> return {x+1: x**2}
>> """
>> t1 = Timer('[func(a) for a in data]', setup=setup)
>> t2 = Timer('list(map(func, data))', setup=setup)
>
> This is very different from the original example, about which the OP
> said that map performs badly, and you doubted it.
I didn't read the OP as making a specific claim about these two *specific*
map and filter examples:
lst = map (lambda x: x*5, lst)
lst = filter (lambda x: x%3 == 1, lst)
I read these as mere examples of a general claim that map and
filter "perform especially bad in CPython compared to a comprehension".
But just for the exercise, I repeated my benchmarks with these specific
examples, comparing:
list(map(lambda x: x*5, data))
[x*5 for x in data]
and
list(filter(lambda x: x%3 == 1, data))
[x for x in data if x%3 == 1]
and again got a roughly factor of two performance difference, with the list
comp being faster. I don't think that justifies the claim of "especially
bad", which to me implies something much worse. If you're going to describe
a mere factor of two as "especially bad", what words do we have left for
something that is ten thousand times slower?
As the wisest man in the universe once said, hyperbole is the most terrible,
awful crime against humanity.
*wink*
[...]
> Thing is, this is extremely common. How often do you actually use a
> comprehension with something that is absolutely exactly a function
> call on the element in question?
"This" being something that can be in-lined in the body of the list comp.
Sure. I cheerfully acknowledge that list comps where you can write an
in-line expression are very common. That's the beauty of list comps!
[...]
> But this conclusion I agree with. There is a performance difference,
> but it is not overly significant. Compared to the *actual work*
> involved in the task (going through one list and doing some operation
> on each operation), the difference between map and a comprehension is
> generally going to be negligible.
I wouldn't go quit
Re: Pre-pep discussion material: in-place equivalents to map and filter
> If slice assignment is done as I hope it will optimize remain memory
operations.
Bad news.
http://stackoverflow.com/questions/4948293/python-slice-assignment-memory-usage/4948508#4948508
> If you want something like C++ move semantics, use C++.
I don't see anything like this in my proposal. If any in-place operation is
"C++ semantics" how do you explain there is already a bunch of in-place
operator stuffed in Python that have even less justification to their
existance than map or filter does such as sort/sorted, reverse/reversed ?
Especially troubling since the optimisation in a sort operation is likely
to be less significant than in a linear algorithm.
2016-11-04 1:03 GMT+01:00 Terry Reedy :
> On 11/3/2016 2:56 AM, arthurhavli...@gmail.com wrote:
>
> lst = [ item for item in lst if predicate(item) ]
>> lst = [ f(item) for item in lst ]
>>
>> Both these expressions feature redundancy, lst occurs twice and item at
>> least twice. Additionally, the readability is hurt, because one has to dive
>> through the semantics of the comprehension to truely understand I am
>> filtering the list or remapping its values.
>>
> ...
>
>> A language support for these operations to be made in-place could improve
>> the efficiency of this operations through reduced use of memory.
>>
>
> We already have that: slice assignment with an iterator.
>
> lst[:] = (item for item in list if predicate(item))
> lst[:] = map(f, lst) # iterator in 3.x.
>
> To save memory, stop using unneeded temporary lists and use iterators
> instead. If slice assignment is done as I hope it will optimize remain
> memory operations. (But I have not read the code.) It should overwrite
> existing slots until either a) the iterator is exhausted or b) existing
> memory is used up. When lst is both source and destination, only case a)
> can happen. When it does, the list can be finalized with its new contents.
>
> As for timings.
>
> from timeit import Timer
> setup = """data = list(range(1))
> def func(x):
> return x
> """
> t1a = Timer('data[:] = [func(a) for a in data]', setup=setup)
> t1b = Timer('data[:] = (func(a) for a in data)', setup=setup)
> t2a = Timer('data[:] = list(map(func, data))', setup=setup)
> t2b = Timer('data[:] = map(func, data)', setup=setup)
>
> print('t1a', min(t1a.repeat(number=500, repeat=7)))
> print('t1b', min(t1b.repeat(number=500, repeat=7)))
> print('t2a', min(t2a.repeat(number=500, repeat=7)))
> print('t2b', min(t2b.repeat(number=500, repeat=7)))
> #
> t1a 0.5675313005414555
> t1b 0.7034254675598604
> t2a 0.518128598520
> t2b 0.5196112759726024
>
> If f does more work, the % difference among these will decrease.
>
>
>
> --
> Terry Jan Reedy
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
On 11/3/2016 2:56 AM, arthurhavli...@gmail.com wrote:
lst = [ item for item in lst if predicate(item) ]
lst = [ f(item) for item in lst ]
Both these expressions feature redundancy, lst occurs twice and item at least
twice. Additionally, the readability is hurt, because one has to dive through
the semantics of the comprehension to truely understand I am filtering the list
or remapping its values.
...
A language support for these operations to be made in-place could improve the
efficiency of this operations through reduced use of memory.
We already have that: slice assignment with an iterator.
lst[:] = (item for item in list if predicate(item))
lst[:] = map(f, lst) # iterator in 3.x.
To save memory, stop using unneeded temporary lists and use iterators
instead. If slice assignment is done as I hope it will optimize remain
memory operations. (But I have not read the code.) It should overwrite
existing slots until either a) the iterator is exhausted or b) existing
memory is used up. When lst is both source and destination, only case
a) can happen. When it does, the list can be finalized with its new
contents.
As for timings.
from timeit import Timer
setup = """data = list(range(1))
def func(x):
return x
"""
t1a = Timer('data[:] = [func(a) for a in data]', setup=setup)
t1b = Timer('data[:] = (func(a) for a in data)', setup=setup)
t2a = Timer('data[:] = list(map(func, data))', setup=setup)
t2b = Timer('data[:] = map(func, data)', setup=setup)
print('t1a', min(t1a.repeat(number=500, repeat=7)))
print('t1b', min(t1b.repeat(number=500, repeat=7)))
print('t2a', min(t2a.repeat(number=500, repeat=7)))
print('t2b', min(t2b.repeat(number=500, repeat=7)))
#
t1a 0.5675313005414555
t1b 0.7034254675598604
t2a 0.518128598520
t2b 0.5196112759726024
If f does more work, the % difference among these will decrease.
--
Terry Jan Reedy
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
arthurhavli...@gmail.com writes: > I would gladly appreciate your returns on this, regarding: > 1 - Whether a similar proposition has been made > 2 - If you find this of any interest at all > 3 - If you have a suggestion for improving the proposal Bleccch. Might be ok as a behind-the-scenes optimization by the compiler. If you want something like C++ move semantics, use C++. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
On Fri, Nov 4, 2016 at 4:00 AM, Steve D'Aprano
wrote:
> On Fri, 4 Nov 2016 01:05 am, Chris Angelico wrote:
>
>> On Thu, Nov 3, 2016 at 7:29 PM, Steven D'Aprano
>> wrote:
>>>> lst = map (lambda x: x*5, lst)
>>>> lst = filter (lambda x: x%3 == 1, lst)
>>>> And perform especially bad in CPython compared to a comprehension.
>>>
>>> I doubt that.
>>>
>>
>> It's entirely possible. A list comp involves one function call (zero
>> in Py2), but map/lambda involves a function call per element in the
>> list. Function calls have overhead.
>
> I don't know why you think that list comps don't involve function calls.
List comps themselves involve one function call (zero in Py2). What
you do inside the expression is your business. Do you agree that list
comps don't have the overhead of opening and closing files?
files = "/etc/hostname", "/etc/resolv.conf", ".bashrc"
contents = [open(fn).read() for fn in files]
In a comparison between comprehensions and map, this cost is
irrelevant, unless your point is that "they're all pretty quick".
> Here's some timing results using 3.5 on my computer. For simplicity, so
> folks can replicate the test themselves, here's the timing code:
>
>
> from timeit import Timer
> setup = """data = list(range(1))
> def func(x): # simulate some calculation
> return {x+1: x**2}
> """
> t1 = Timer('[func(a) for a in data]', setup=setup)
> t2 = Timer('list(map(func, data))', setup=setup)
This is very different from the original example, about which the OP
said that map performs badly, and you doubted it. In that example, the
list comp includes the expression directly, whereas map (by its
nature) must use a function. The "inline expression" of a
comprehension is more efficient than the "inline expression" of a
lambda function given to map.
> And here's the timing results on my machine:
>
> py> min(t1.repeat(number=1000, repeat=7)) # list comp
> 18.2571472954005
> py> min(t2.repeat(number=1000, repeat=7)) # map
> 18.157311914488673
>
> So there's hardly any difference, but map() is about 0.5% faster in this
> case.
Right. As has often been stated, map is perfectly efficient, *if* the
body of the comprehension would be simply a function call, nothing
more. You can map(str, stuff) to stringify a bunch of things. Nice.
But narrow, and not the code that was being discussed.
> Now, it is true that *if* you can write the list comprehension as a direct
> calculation in an expression instead of a function call:
>
> [a+1 for a in data]
>
> *and* compare it to map using a function call:
>
> map(lambda a: a+1, data)
>
>
> then the overhead of the function call in map() may be significant.
Thing is, this is extremely common. How often do you actually use a
comprehension with something that is absolutely exactly a function
call on the element in question?
> But the
> extra cost is effectively just a multiplier. It isn't going to change
> the "Big Oh" behaviour.
Sure it doesn't. In each case, the cost is linear. But the work is
linear, so I would expect the time to be linear.
> So map() here is less than a factor of two slower. I wouldn't call
> that "especially bad" -- often times, a factor of two is not important.
> What really hurts is O(N**2) performance, or worse.
>
> So at worst, map() is maybe half as fast as a list comprehension, and at
> best, its perhaps a smidgen faster. I would expect around the same
> performance, differing only by a small multiplicative factor: I wouldn't
> expect one to be thousands or even tens of times slower that the other.
But this conclusion I agree with. There is a performance difference,
but it is not overly significant. Compared to the *actual work*
involved in the task (going through one list and doing some operation
on each operation), the difference between map and a comprehension is
generally going to be negligible.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
I understand that, the cost of change is such that it's very unlikely something like this ever goes into Python, but I feel like the interest of the proposition is being underestimated here, that's why I'm going to argue a few points and give a bit more context as needed. > While mapping and filtering are common operations, I'm not sure mapping > and filtering and then reassigning back to the original sequence is > especially common. It depends of your context. On the last 3 months, I stumbled across this at least 3 times, which is 3 times more than I used a lambda or a metaclass or a decorator or other fancy language feature that we simply avoid whenever possible. It also happened to my collegue. I remember these examples because we had a bit of humour about how nice can be inlined ternaries inside comprehensions, but I could be missing a lot more. The reason I'm being especially impacted by this is because I am maintainer of a decent-sized Python application (~50-100K lines of code) that extensively uses lists and dictionaries. We value "low level" data manipulation and efficiency a lot more than complex, non-obvious constructs. In other contexts, it may be very different. Note that my context is only relevant for illustration here, I don't expect a feature to save me since we are currently shipping to Centos 6 and thus will not see the light of Python 3.7 in the next 10 years (optimistic estimation). > Arthur, I would suggest looking at what numpy and pandas do. In my example context, their benefits can't apply, because I'm not going to rewrite code that uses lists for them to uses np.array instead for example. Although the performance boost is likely to be bigger if used properly, I would have prefered a lesser boost that comes for (almost) free. Like most Python programmers, I'm not in the case of needing a performance boost very bad, but that does not mean I disregard performance entirely. The need of performance is not so binary that it either don't matter at all or is enough to motivate a rewrite. > To my eyes, this proposed syntax is completely foreign I must admit I don't have much imagination for syntax proposals...all that mattered to me here was to make it clear you are doing an in-place modification. Feel free to completely ignore that part. Any proposal welcomed of course. About Readability & Redundancy I have misused the terms here, but I wasn't expecting so much nitpicking. I should have used the term maintenability, because that one is bland and subjective enough that nobody would have noticed :D How about "I find that cooler." Good enough ? In a less sarcastic tone: What I truely meant here is that when you contain the behavior of your code inside a specific keyword or syntax, you are making your intentions clear to the reader. It may be harder for him to gain access to the knowledge in the first place, but makes it easier over time. Famous example: When you learned programming, you may have had no idea what "+=" was doing, but now you do, you probably rate "a += 2" syntax to be much better than "a = a + 2". You make the economy of a token, but more important, you make your intentions clearer because "+=" rings a bell, wihle "=" is a more generic syntax with a broader meaning. > So map() here is less than a factor of two slower. I wouldn't call > that "especially bad" -- often times, a factor of two is not important. > What really hurts is O(N**2) performance, or worse. When you evaluate your application bottleneck or your average daily algorithmic evaluation, perhaps. When regarding language core features we are not on the same scale. If a language X is 2 times faster than a language Y to do the same task, that's a huge seller, and is of real importance. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
On Fri, 4 Nov 2016 01:05 am, Chris Angelico wrote:
> On Thu, Nov 3, 2016 at 7:29 PM, Steven D'Aprano
> wrote:
>>> lst = map (lambda x: x*5, lst)
>>> lst = filter (lambda x: x%3 == 1, lst)
>>> And perform especially bad in CPython compared to a comprehension.
>>
>> I doubt that.
>>
>
> It's entirely possible. A list comp involves one function call (zero
> in Py2), but map/lambda involves a function call per element in the
> list. Function calls have overhead.
I don't know why you think that list comps don't involve function calls.
Here's some timing results using 3.5 on my computer. For simplicity, so
folks can replicate the test themselves, here's the timing code:
from timeit import Timer
setup = """data = list(range(1))
def func(x): # simulate some calculation
return {x+1: x**2}
"""
t1 = Timer('[func(a) for a in data]', setup=setup)
t2 = Timer('list(map(func, data))', setup=setup)
And here's the timing results on my machine:
py> min(t1.repeat(number=1000, repeat=7)) # list comp
18.2571472954005
py> min(t2.repeat(number=1000, repeat=7)) # map
18.157311914488673
So there's hardly any difference, but map() is about 0.5% faster in this
case.
Now, it is true that *if* you can write the list comprehension as a direct
calculation in an expression instead of a function call:
[a+1 for a in data]
*and* compare it to map using a function call:
map(lambda a: a+1, data)
then the overhead of the function call in map() may be significant. But the
extra cost is effectively just a multiplier. It isn't going to change
the "Big Oh" behaviour. Here's an example:
t3 = Timer('[a+1 for a in data]', setup=setup)
t4 = Timer('list(map(lambda a: a+1, data))', setup=setup)
extra = """from functools import partial
from operator import add
"""
t5 = Timer('list(map(partial(add, 1), data))', setup=setup+extra)
And the timing results:
py> min(t3.repeat(number=1000, repeat=7)) # list comp with expression
2.6977453008294106
py> min(t4.repeat(number=1000, repeat=7)) # map with function
4.280411267653108
py> min(t5.repeat(number=1000, repeat=7)) # map with partial
3.446241058409214
So map() here is less than a factor of two slower. I wouldn't call
that "especially bad" -- often times, a factor of two is not important.
What really hurts is O(N**2) performance, or worse.
So at worst, map() is maybe half as fast as a list comprehension, and at
best, its perhaps a smidgen faster. I would expect around the same
performance, differing only by a small multiplicative factor: I wouldn't
expect one to be thousands or even tens of times slower that the other.
--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
On 11/3/2016 4:29 AM, Steven D'Aprano wrote: Nonsense. It is perfectly readable because it is explicit about what is being done, unlike some magic method that you have to read the docs to understand what it does. Agreed. A list comprehension or for-loop is more general and can be combined so you can do both: alist[:] = [func(x) for x in alist if condition(x)] The qualifier 'list' is not needed. The right hand side of slice assignment just has to be an iterable. So a second interation to build an intermediate list is not required. alist[:] = (func(x) for x in alist if condition(x)) The parentheses around the generator expression are required here. (Steven, I know you know that, but not everyone else will.) -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
On Thu, Nov 3, 2016 at 7:29 PM, Steven D'Aprano wrote: >> lst = map (lambda x: x*5, lst) >> lst = filter (lambda x: x%3 == 1, lst) >> And perform especially bad in CPython compared to a comprehension. > > I doubt that. > It's entirely possible. A list comp involves one function call (zero in Py2), but map/lambda involves a function call per element in the list. Function calls have overhead. Arthur, I would suggest looking at what numpy and pandas do. When you're working with ridiculously large data sets, they absolutely shine; and if you're not working with that much data, the performance of map or a list comp is unlikely to be significant. If the numpy folks have a problem that can't be solved without new syntax, then a proposal can be brought to the core (like matmul, which was approved and implemented). ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Pre-pep discussion material: in-place equivalents to map and filter
Hi everybody, I have an enhancement proposal for Python and, as suggested by PEP 1, am exposing a stub to the mailing list before possibly starting writing a PEP. This is my first message to python mailing list. I hope you will find this content of interest. Python features a powerful and fast way to create lists through comprehensions. Because of their ease of use and efficiency through native implementation, they are an advantageous alternative to map, filter, and more. However, when used in replacement for an in-place version of these functions, they are sub-optimal, and Python offer no alternative. This lack of implementation have already been pointed out: http://stackoverflow.com/questions/4148375/is-there-an-in-place-equivalent-to-map-in-python Notice the concerns of OPs in his comments to replies in this one: http://stackoverflow.com/questions/3000461/python-map-in-place In this answer, developpers here are wondering about performance issues regarding both loop iteration and comprehension: http://stackoverflow.com/a/1540069/3323394 I would suggest to implement a language-level, in-place version for them. There is severeal motivations for this: 1 - Code readability and reduced redundancy lst = [ item for item in lst if predicate(item) ] lst = [ f(item) for item in lst ] Both these expressions feature redundancy, lst occurs twice and item at least twice. Additionally, the readability is hurt, because one has to dive through the semantics of the comprehension to truely understand I am filtering the list or remapping its values. Map and filter, although they are more explicit, also feature redundancy. They look OK with functional predicate: lst = map (f, lst) lst = filter (predicate, lst) But are less elegant when using an expression, than one has to convert through a lambda: lst = map (lambda x: x*5, lst) lst = filter (lambda x: x%3 == 1, lst) And perform especially bad in CPython compared to a comprehension. 2 - Efficiency A language support for these operations to be made in-place could improve the efficiency of this operations through reduced use of memory. I would propose this syntax. (TODO: find appropriate keywords I guess): lst.map x: x*5 lst.filter x: x%3 == 1 They can work for dictionaries too. dct.map k,v: v*5 dct.filter k,v: k+v == 10 The reasonning for the need of a language-level approach is the need for an efficient implementation that would support giving an arbitrary expression and not only a function. Expressions are shorter and, I guess, would be more efficient. I would gladly appreciate your returns on this, regarding: 1 - Whether a similar proposition has been made 2 - If you find this of any interest at all 3 - If you have a suggestion for improving the proposal Thanks for reading. Have a nice day -- https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
On Thursday, November 3, 2016 at 4:30:00 AM UTC-4, Steven D'Aprano wrote: > On Thursday 03 November 2016 17:56, arthurhavli...@gmail.com wrote: > > I would propose this syntax. (TODO: find appropriate keywords I guess): > > > > lst.map x: x*5 > > lst.filter x: x%3 == 1 > > I think the chances of Guido accepting new syntax for something as trivial as > this with three existing solutions is absolutely zero. > > I think the chances of Guido accepting new list/dict methods for in place map > and/or filter is a tiny bit higher than zero. To my eyes, this proposed syntax is completely foreign. "lst.map" looks like attribute or method access on lst, but there's no operation on the result, or function call. They implicitly assign back to lst, with no recognizable syntax to indicate that they do (= or "as" is the usual marker). While mapping and filtering are common operations, I'm not sure mapping and filtering and then reassigning back to the original sequence is especially common. It's certainly not common enough to deserve completely new syntax when the existing methods already exist. Part of the problem here is that you value explicitness, but also are trying to reduce redundancy. When you say that lst occurs twice in your examples, what I see is that it occurs twice because it's being used for two different things: it is the source of the data, and it is also the target for the result. I think it is good to have it appear twice in this case, especially since there's no reason to think it will usually be used for both purposes. How do I do exactly the same data manipulation, but then assign it to lst2 because I belatedly realized I wanted both the before and after list? Under your proposed syntax, I would have to completely rewrite the line to use a different filtering mechanism because now I need to unbundle the filtering and the assignment. That seems unfortunate. You've done the right thing by bringing the proposal here. I think it is useful to see how people approach the language, and where they want changes. Discussing the pros and cons is a good way to get a deeper appreciation both for the language and the rationale for its design. But I don't think this proposal has a chance of moving forward. --Ned. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pre-pep discussion material: in-place equivalents to map and filter
On Thursday 03 November 2016 17:56, arthurhavli...@gmail.com wrote: [...] > Python features a powerful and fast way to create lists through > comprehensions. Because of their ease of use and efficiency through native > implementation, they are an advantageous alternative to map, filter, and > more. However, when used in replacement for an in-place version of these > functions, they are sub-optimal, and Python offer no alternative. Of course Python offers an alternative: the basic for loop. for i, x in enumerate(alist): alist[i] = func(x) Not enough of a one-liner for you? Write a helper function: def inplace_map(func, alist): for i, x in enumerate(alist): alist[i] = func(x) In practice, you may even find that except for the most enormous lists (so big that memory becomes an issue, so we're talking tens of millions of items) it will probably be faster to use a slice and a comprehension: alist[:] = [func(x) for x in alist] [...] > Notice the concerns of OPs in his comments to replies in this one: > http://stackoverflow.com/questions/3000461/python-map-in-place What I saw was the OP's *premature optimization*: "I wanted to use map for performance gains" "I figured map would be quicker than the list comprehension way" Although in fairness he does also say: "I'm working on a big list, and the times we're talking about are seconds in difference, which clearly matters" I'm not necessarily sure that seconds always matters -- if your code takes 90 seconds, or 96 seconds, who is going to even notice? But let's assume he's right and a few seconds difference makes a real difference. He has three obvious alternatives to waiting until Python 3.7 (the earliest such a new feature can be added): - modify the list in place with a for-loop; - slice assignment using map; - slice assignment using list comprehension. > 1 - Code readability and reduced redundancy > > lst = [ item for item in lst if predicate(item) ] > lst = [ f(item) for item in lst ] > > Both these expressions feature redundancy, lst occurs twice and item at least > twice. That's not what redundancy means. "Redundancy" doesn't refer to the re-use of any arbitrary token. It means doing the same thing twice in two different places. > Additionally, the readability is hurt, because one has to dive through > the semantics of the comprehension to truely understand I am filtering the > list or remapping its values. Nonsense. It is perfectly readable because it is explicit about what is being done, unlike some magic method that you have to read the docs to understand what it does. A list comprehension or for-loop is more general and can be combined so you can do both: alist[:] = [func(x) for x in alist if condition(x)] > Map and filter, although they are more explicit, *Less* explicit. To most people, "map" means the thing that you follow when you are travelling in unfamiliar territory, and "filter" means the paper doohickey you put in your coffee machine to keep the ground up coffee from ending up in your cup. > also feature redundancy. > They look OK with functional predicate: > > lst = map (f, lst) > lst = filter (predicate, lst) > > But are less elegant when using an expression, than one has to convert > through a lambda: > > lst = map (lambda x: x*5, lst) > lst = filter (lambda x: x%3 == 1, lst) And that's why we have list comprehensions. > And perform especially bad in CPython compared to a comprehension. I doubt that. > 2 - Efficiency > > A language support for these operations to be made in-place could improve the > efficiency of this operations through reduced use of memory. *shrug* Saving memory sometimes costs time. > I would propose this syntax. (TODO: find appropriate keywords I guess): > > lst.map x: x*5 > lst.filter x: x%3 == 1 I think the chances of Guido accepting new syntax for something as trivial as this with three existing solutions is absolutely zero. I think the chances of Guido accepting new list/dict methods for in place map and/or filter is a tiny bit higher than zero. > The reasonning for the need of a language-level approach is the need for an > efficient implementation that would support giving an arbitrary expression > and not only a function. We already have three of those: for-loops, list comprehensions, and map. -- Steven 299792.458 km/s — not just a good idea, it’s the law! -- https://mail.python.org/mailman/listinfo/python-list
Re: Create a map for data to flow through
Sayth Renshaw wrote: Is there a standard library feature that allows you to define a declarative map or statement that defines the data and its objects to be parsed and output format? Not really. Just wondering as for loops are good but when i end up 3-4 for loops deep and want multiple matches at each level i am finding it harder to manage. When your loops get more than about one or two levels deep, think about separaring each level into its own function. That will allow you to think about just one part of the problem at a time, and help you to not get confused. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Create a map for data to flow through
Is there a standard library feature that allows you to define a declarative map or statement that defines the data and its objects to be parsed and output format? Just wondering as for loops are good but when i end up 3-4 for loops deep and want multiple matches at each level i am finding it harder to manage. I am reading here https://docs.python.org/3/howto/functional.html Is there a better way than loops on loops on loops etc? Thinking that for loop is quick at the start but there probably is a more direct way which while slower may be clearer over the long run. Cheers Sayth -- https://mail.python.org/mailman/listinfo/python-list
Re: which library has map reduce and how to use it for this case
> > On Friday, June 10, 2016 at 10:04:09 PM UTC+8, Michael Selik wrote: > > You'll need to explain the problem in more detail. Instead of talking > about operators and columns, what is the actual, real-world problem? On Sat, Jun 11, 2016 at 2:21 AM meInvent bbird wrote: > there are six operator, and a logic table initial in b variable > I wasn't asking about the algorithm. I was asking what the task is. If you explain your research question that will help me (and I assume, the rest of the mailing list) understand your problem better. Maybe we can suggest a better algorithm. Give us some context: What data is being input? What is the desired output? What decision will be made based on this calculation? -- https://mail.python.org/mailman/listinfo/python-list
Re: which library has map reduce and how to use it for this case
there are six operator, and a logic table initial in b variable
aa[b[j][i]]
[aa[b[i][0:1]+b[i][2:3]
are just like vlookup to find output of each operator
acting on first column and second column, second column and third column
, first column and third column
and searching a output columns which is result sum is 27*2
and record the path if succeed,
it record the path and output any path which has result is 27*2 described
before
op1(op2(op3(op1(op2(),),op1(op2(),))), op1(op2(),))
there are two cases, first cases b are logic table
later case are for six operators acting on the result column from previous
result before recursive call
def DFS(b, deep, maxx, sourceoperators, path):
initlist = []
if deep > 0:
print("deep=", deep)
for aa,bb in itertools.combinations(sourceoperators, 2):
print(aa,bb)
if deep == maxx:
finalresult = []
op1xy = [aa[b[i][0:1]+b[i][1:2]] for i in range(len(b))]
op1yz = [aa[b[i][1:2]+b[i][2:3]] for i in range(len(b))]
op1xz = [aa[b[i][0:1]+b[i][2:3]] for i in range(len(b))]
op2xy = [bb[b[i][0:1]+b[i][1:2]] for i in range(len(b))]
op2yz = [bb[b[i][1:2]+b[i][2:3]] for i in range(len(b))]
op2xz = [bb[b[i][0:1]+b[i][2:3]] for i in range(len(b))]
if sum(op1xy) == 54:
path.append([(deep, aa, b, "xy")])
else:
initlist.append(op1xy)
if sum(op1yz) == 54:
path.append([(deep, aa, b, "yz")])
else:
initlist.append(op1yz)
if sum(op1xz) == 54:
path.append([(deep, aa, b, "xz")])
else:
initlist.append(op1xz)
if sum(op2xy) == 54:
path.append([(deep, bb, b, "xy")])
else:
initlist.append(op2xy)
if sum(op2yz) == 54:
path.append([(deep, bb, b, "yz")])
else:
initlist.append(op2yz)
if sum(op2xz) == 54:
path.append([(deep, bb, b, "xz")])
else:
initlist.append(op2xz)
else:
level = []
for j in range(len(b)):
op1xy = [aa[b[j][i]] for i in range(len(b[j]))]
op2xy = [bb[b[j][i]] for i in range(len(b[j]))]
if sum(op1xy) == 54:
path.append([(deep, aa, b[j], "xy")])
else:
initlist.append(op1xy)
if sum(op2xy) == 54:
path.append([(deep, bb, b[j], "xy")])
else:
initlist.append(op2xy)
level.extend([op1xy, op2xy])
if deep == maxx:
b = []
print("initlist=")
print(len(initlist))
for aaa,bbb in itertools.combinations(initlist, 2):
b.append([str(i)+str(j) for i,j in zip(aaa, bbb)])
if deep > 0:
path2 = DFS(b, deep-1, maxx, sourceoperators, path)
path.append(path2)
return path
path = []
mresult = DFS(b, 2, 2, mylist, path)
On Friday, June 10, 2016 at 10:04:09 PM UTC+8, Michael Selik wrote:
> On Thu, Jun 9, 2016, 9:11 PM Ho Yeung Lee wrote:
>
> > input are these six operators, output is finding full column of 27
> > elements add together is 54
> >
> > i got memory error when searching this,
> >
> > i have difficulty in understanding map reduce and transforming my program
> > into map reduce problem
> >
>
> Why do you think you need map-reduce?
>
> You'll need to explain the problem in more detail. Instead of talking about
> operators and columns, what is the actual, real-world problem?
>
> >
--
https://mail.python.org/mailman/listinfo/python-list
Re: which library has map reduce and how to use it for this case
On Thu, Jun 9, 2016, 9:11 PM Ho Yeung Lee wrote: > input are these six operators, output is finding full column of 27 > elements add together is 54 > > i got memory error when searching this, > > i have difficulty in understanding map reduce and transforming my program > into map reduce problem > Why do you think you need map-reduce? You'll need to explain the problem in more detail. Instead of talking about operators and columns, what is the actual, real-world problem? > -- https://mail.python.org/mailman/listinfo/python-list
Re: which library has map reduce and how to use it for this case
input are these six operators, output is finding full column of 27 elements add
together is 54
i got memory error when searching this,
i have difficulty in understanding map reduce and transforming my program into
map reduce problem
M1 = {}
M2 = {}
M3 = {}
M4 = {}
M5 = {}
V6 = {}
M1['00']=0
M1['01']=2
M1['02']=1
M1['10']=1
M1['11']=1
M1['12']=1
M1['20']=1
M1['21']=1
M1['22']=2
M2['00']=0
M2['01']=1
M2['02']=1
M2['10']=1
M2['11']=1
M2['12']=1
M2['20']=1
M2['21']=1
M2['22']=1
M3['00']=2
M3['01']=2
M3['02']=2
M3['10']=0
M3['11']=2
M3['12']=1
M3['20']=0
M3['21']=1
M3['22']=2
M4['00']=1
M4['01']=2
M4['02']=1
M4['10']=2
M4['11']=2
M4['12']=2
M4['20']=0
M4['21']=1
M4['22']=2
M5['00']=0
M5['01']=1
M5['02']=1
M5['10']=0
M5['11']=2
M5['12']=1
M5['20']=0
M5['21']=1
M5['22']=1
V6['00']=1
V6['01']=1
V6['02']=2
V6['10']=1
V6['11']=2
V6['12']=1
V6['20']=1
V6['21']=2
V6['22']=2
MM = {}
MM[0] = M1
MM[1] = M2
MM[2] = M3
MM[3] = M4
MM[4] = M5
MM[5] = V6
m = 3
b = [str(i)+str(j)+str(k) for i in range(m) for j in range(m) for k in range(m)]
import itertools
deep = 3
final = []
mylist = [MM[i] for i in range(0,7-1)]
b = [str(i)+str(j)+str(k) for i in range(m) for j in range(m) for k in range(m)]
def DFS(b, deep, maxx, sourceoperators, path):
initlist = []
if deep > 0:
print("deep=", deep)
for aa,bb in itertools.combinations(sourceoperators, 2):
print(aa,bb)
if deep == maxx:
finalresult = []
op1xy = [aa[b[i][0:1]+b[i][1:2]] for i in range(len(b))]
op1yz = [aa[b[i][1:2]+b[i][2:3]] for i in range(len(b))]
op1xz = [aa[b[i][0:1]+b[i][2:3]] for i in range(len(b))]
op2xy = [bb[b[i][0:1]+b[i][1:2]] for i in range(len(b))]
op2yz = [bb[b[i][1:2]+b[i][2:3]] for i in range(len(b))]
op2xz = [bb[b[i][0:1]+b[i][2:3]] for i in range(len(b))]
if sum(op1xy) == 54:
path.append([(deep, aa, "xy")])
if sum(op1yz) == 54:
path.append([(deep, aa, "yz")])
if sum(op1xz) == 54:
path.append([(deep, aa, "xz")])
if sum(op2xy) == 54:
path.append([(deep, bb, "xy")])
if sum(op2yz) == 54:
path.append([(deep, bb, "yz")])
if sum(op2xz) == 54:
path.append([(deep, bb, "xz")])
initlist.append(op1xy)
initlist.append(op1yz)
initlist.append(op1xz)
initlist.append(op2xy)
initlist.append(op2yz)
initlist.append(op2xz)
else:
level = []
for j in range(len(b)):
op1xy = [aa[b[j][i]] for i in range(len(b[j]))]
op2xy = [bb[b[j][i]] for i in range(len(b[j]))]
if sum(op1xy) == 54:
path.append([(deep, aa, "xy")])
if sum(op2xy) == 54:
path.append([(deep, bb, "xy")])
level.append(op1xy)
level.append(op2xy)
initlist.append(op1xy)
initlist.append(op2xy)
if deep == maxx:
b = []
#print(len(list(itertools.combinations(initlist, 2
for aaa,bbb in itertools.combinations(initlist, 2):
b.append([str(i)+str(j) for i,j in zip(aaa, bbb)])
path = DFS(b, deep-1, maxx, sourceoperators, path)
else:
#print(len(list(itertools.combinations(initlist, 2
for aaa,bbb in itertools.combinations(initlist, 2):
b.append([str(i)+str(j) for i,j in zip(aaa, bbb)])
path = DFS(b, deep-1, maxx, sourceoperators, path)
return path
path = []
mresult = DFS(b, 2, 2, mylist, path)
Michael Selik於 2016年6月10日星期五 UTC+8上午6時45分14秒寫道：
> I like using Yelp's mrjob module (https://github.com/Yelp/mrjob) to run
> Python on Hadoop.
>
> On Thu, Jun 9, 2016 at 2:56 AM Ho Yeung Lee
> wrote:
>
> > [... a bunch of code ...]
>
>
> If you want to describe a map-reduce problem, start with the data. What
> does a record of your input data look like?
>
> Then think about your mapper. What key-value pairs will you extract from
> each line of data?
>
> Then think about your reducer. For a single key and its associated values,
> what will you calculate?
--
https://mail.python.org/mailman/listinfo/python-list
Re: which library has map reduce and how to use it for this case
I like using Yelp's mrjob module (https://github.com/Yelp/mrjob) to run Python on Hadoop. On Thu, Jun 9, 2016 at 2:56 AM Ho Yeung Lee wrote: > [... a bunch of code ...] If you want to describe a map-reduce problem, start with the data. What does a record of your input data look like? Then think about your mapper. What key-value pairs will you extract from each line of data? Then think about your reducer. For a single key and its associated values, what will you calculate? -- https://mail.python.org/mailman/listinfo/python-list
which library has map reduce and how to use it for this case
i got M1 to M5 and V6 operator
and would like to do a full combination and result will also act among each
other,
map reduce attract my application
how to use this in this example?
actually below is like vlookup
then example is op3(op2(op1(x->y, y->z)), x->z)
search which combinations will result in a column result is 1
M1 = {}
M2 = {}
M3 = {}
M4 = {}
M5 = {}
V6 = {}
M1['00']=0
M1['01']=1
M1['02']=1
M1['10']=1
M1['11']=1
M1['12']=1
M1['20']=1
M1['21']=1
M1['22']=1
M2['00']=0
M2['01']=0
M2['02']=1
M2['10']=0
M2['11']=1
M2['12']=1
M2['20']=1
M2['21']=1
M2['22']=1
M3['00']=0
M3['01']=0
M3['02']=0
M3['10']=0
M3['11']=1
M3['12']=1
M3['20']=0
M3['21']=1
M3['22']=1
M4['00']=0
M4['01']=1
M4['02']=0
M4['10']=1
M4['11']=1
M4['12']=1
M4['20']=0
M4['21']=1
M4['22']=1
M5['00']=0
M5['01']=0
M5['02']=0
M5['10']=0
M5['11']=1
M5['12']=1
M5['20']=0
M5['21']=1
M5['22']=1
V6['00']=1
V6['01']=1
V6['02']=1
V6['10']=1
V6['11']=1
V6['12']=1
V6['20']=0
V6['21']=1
V6['22']=1
MM = {}
MM[0] = M1
MM[1] = M2
MM[2] = M3
MM[3] = M4
MM[4] = M5
MM[5] = V6
m = 3
b = [str(i)+str(j)+str(k) for i in range(m) for j in range(m) for k in range(m)]
b[21][0:1]+b[21][1:2]
b[21][1:2]+b[21][2:3]
b[21][0:1]+b[21][2:3]
#def node(mmm, A):
#H2 = [MM[mmm][b[i][0:1]+b[i][1:2]] for i in range(len(b))]
#return H2
#node(5, b[i][0:1]+b[i][1:2])
H2 = [MM[5][b[i][0:1]+b[i][1:2]] for i in range(len(b))]
H3 = [MM[5][b[i][0:1]+b[i][1:2]] for i in range(len(b))]
[str(k)+str(v) for k, v in zip(H2, H3)]
import itertools
deep = 3
finalresult = []
def DFS(H2, H3, b, deep, mresult)
mylist = [i for i in range(0,7-1)]
for aa,bb in itertools.combinations(mylist, 2):
print(aa,bb)
if deep == 3:
#op0, op1
op1xy = [MM[aa][b[i][0:1]+b[i][1:2]] for i in range(len(b))]
op1yz = [MM[aa][b[i][1:2]+b[i][2:3]] for i in range(len(b))]
op1xz = [MM[aa][b[i][0:1]+b[i][2:3]] for i in range(len(b))]
op2xy = [MM[bb][b[i][0:1]+b[i][1:2]] for i in range(len(b))]
op2yz = [MM[bb][b[i][1:2]+b[i][2:3]] for i in range(len(b))]
op2xz = [MM[bb][b[i][0:1]+b[i][2:3]] for i in range(len(b))]
mresult.append(op1xy)
mresult.append(op1yz)
mresult.append(op1xz)
mresult.append(op2xy)
mresult.append(op2yz)
mresult.append(op2xz)
else:
H1 = H2
H9 = H3
ba = b
b = [str(k)+str(v) for k, v in zip(H2, H3)]
DFS(H1, H9, b)
b = ba
DFS(0, 0, 0, 3, finalresult)
--
https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On Friday, April 1, 2016 at 1:58:29 PM UTC+5:30, Tim Golden wrote: > > For the latter, I take the view that I know where the delete key is (or > the "ignore thread" button or whatever) and I just skip the thread when > it shows up. > Feel free to contact the list owner [python list-owner] if > you think there's a real contravention of the list etiquette but I'm > personally not inclined to jump too heavily on rambling discussions or > wild-eyed ideas as such. I thought sanity was frowned upon out here -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
Steven D'Aprano : > On Fri, 1 Apr 2016 07:27 pm, Tim Golden wrote: > >> FWIW I'm broadly with Antoon here: wider-ranging discussions can be >> interesting and useful. > > Sure. But sometimes conversations are going nowhere: That's why GNUS has the "k" command to wipe out a whole thread. I know, it depends on threading working... Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On Fri, 1 Apr 2016 07:27 pm, Tim Golden wrote: > FWIW I'm broadly with Antoon here: wider-ranging discussions can be > interesting and useful. Sure. But sometimes conversations are going nowhere: http://www.youtube.com/watch?v=kQFKtI6gn9Y http://www.montypython.net/scripts/argument.php [...] > If it becomes clear that the poster is in fact pushing for a change > (either intelligently thought-out or naively ill-considered) then I > would push them towards Python-ideas sooner than later, because that's > exactly the purpose of *that* mailing list. People on python-ideas want > to go to and fro over the relative merits of proposals. Specifically, I > believe that's the only mailing list which GvR follows apart from > python-dev. Any discussion here would likely have to be repeated over > there anyway, so why not go there earlier on? Informally, ideas are supposed to have an initial "sanity check" here to avoid the really silly ideas: Q: "Suggestion: I'm sick of writing for key, value in other_dict.items(): mydict[key] = value I think that dicts should have a method to copy all the keys and values from another dict." A: "You mean dict.update?" Or perhaps: Q: "I think that object oriented programming is too inefficient. I think that Python should get rid of all the objects and just be a lightweight, easy-to-read wrapper around C, with the same semantics and limitations as the 1988 C standard." A: "Surely you aren't serious?" But most people don't bother passing ideas through here first, they just go straight to Python-Ideas. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On 01/04/2016 08:59, Antoon Pardon wrote: Op 31-03-16 om 16:12 schreef Mark Lawrence via Python-list: On 31/03/2016 14:27, Random832 wrote: So can we discuss how a unified method to get a set of all valid subscripts (and/or subscript-value pairs) on an object would be a useful thing to have without getting bogged down in theoretical claptrap about the meaning of the mapping contract? We can discuss anything here until the cows come home, but it's a complete waste of time if the powers that be over on python-ideas and/or python-dev don't agree. This was suggested a day or two back but seems to have gone completely over people's heads. Just because you are not interested, doesn't mean it's a complete waste of time. Discussions like this often enough produce suggestions on how one could handle these things within python without the need for the powers that be to agree on anything. If you are not interested just don't contribute. Others can make up their own mind on whether this is a waste of their time or not. Who said I'm not interested? It is a simple fact of life in Python world that anything that gets discussed has to go through python-dev, and possibly python-ideas first. You can spend years discussing anything you like here and get 100% agreement, but if the devlopers say no it does not happen. I believe that this proposal is like trying to change the design of the Morris Minor and a McClaren Mercedes because they're both cars, so you can make them similar. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On 01/04/2016 08:59, Antoon Pardon wrote: > Op 31-03-16 om 16:12 schreef Mark Lawrence via Python-list: >> On 31/03/2016 14:27, Random832 wrote: >>> So can we discuss how a unified method to get a set of all valid >>> subscripts (and/or subscript-value pairs) on an object would be a useful >>> thing to have without getting bogged down in theoretical claptrap about >>> the meaning of the mapping contract? >>> >> >> We can discuss anything here until the cows come home, but it's a >> complete waste of time if the powers that be over on python-ideas >> and/or python-dev don't agree. This was suggested a day or two back >> but seems to have gone completely over people's heads. > > Just because you are not interested, doesn't mean it's a complete waste of > time. > Discussions like this often enough produce suggestions on how one could handle > these things within python without the need for the powers that be to agree on > anything. > > If you are not interested just don't contribute. Others can make up their own > mind on whether this is a waste of their time or not. FWIW I'm broadly with Antoon here: wider-ranging discussions can be interesting and useful. (And informative, especially where people speak knowledgeably about an area outside my own competence). There *are* technical forums where anything outside their strict subject matter is frowned upon or curtailed. I don't think we need to be that rigid here. However I think there are a couple of lines which can be crossed. In one case a a poster (perhaps abruptly) says: I think Python should do this; why doesn't it? The other is where the discussion goes so far into cloud-cuckoo land that it alienates all but a few devoted adherents to the thread. Especially where it goes round in circles. For the latter, I take the view that I know where the delete key is (or the "ignore thread" button or whatever) and I just skip the thread when it shows up. Perhaps missing some interesting points in the process if it comes back down to earth but that's the way it goes. For the former, I think it's fine if someone is asking in a genuine spirit of enquiry, ie to learn about the history of a particular decision or the ramifications of an alternative which might not be obvious at first glance. (Why do we have both lists and tuples? Why does Python index from 0? etc). People who know something about it can explain if they wish. If it becomes clear that the poster is in fact pushing for a change (either intelligently thought-out or naively ill-considered) then I would push them towards Python-ideas sooner than later, because that's exactly the purpose of *that* mailing list. People on python-ideas want to go to and fro over the relative merits of proposals. Specifically, I believe that's the only mailing list which GvR follows apart from python-dev. Any discussion here would likely have to be repeated over there anyway, so why not go there earlier on? Courtesy & respect on this and any list are important, so as long as someone's not being genuinely rude or abusive, your best plan is to make your point clearly and politely and then step back. I've been impressed again and again on Python lists where people maintain a courteous front in the course of a perhaps quite heated discussion. And more impressed when people who have lost their cool come back and apologise (without necessarily backing down from their point, of course). Feel free to contact the list owner [python-list-ow...@python.org] if you think there's a real contravention of the list etiquette but I'm personally not inclined to jump too heavily on rambling discussions or wild-eyed ideas as such. TJG -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
> On Mar 31, 2016, at 10:02 AM, Marko Rauhamaa wrote: > > However, weirdly, dicts have get but lists don't. Read PEP 463 for discussion on this topic. https://www.python.org/dev/peps/pep-0463/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
Op 31-03-16 om 16:12 schreef Mark Lawrence via Python-list: > On 31/03/2016 14:27, Random832 wrote: >> So can we discuss how a unified method to get a set of all valid >> subscripts (and/or subscript-value pairs) on an object would be a useful >> thing to have without getting bogged down in theoretical claptrap about >> the meaning of the mapping contract? >> > > We can discuss anything here until the cows come home, but it's a > complete waste of time if the powers that be over on python-ideas > and/or python-dev don't agree. This was suggested a day or two back > but seems to have gone completely over people's heads. Just because you are not interested, doesn't mean it's a complete waste of time. Discussions like this often enough produce suggestions on how one could handle these things within python without the need for the powers that be to agree on anything. If you are not interested just don't contribute. Others can make up their own mind on whether this is a waste of their time or not. -- Antoon. -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On 3/31/2016 10:13 AM, Marko Rauhamaa wrote: One could compose a table of correspondences: with some corrections --- list (L)dict (D) --- L[key] = value D[key] = value del L[key] (*) del L[key] (*) reassigns all keys key >= 0 and key < len(L) key in D '-len(L) <= key < len(L)' or 'key in range(-len(L), len(L)' Lists, tuples, and ranges have 2 keys for each value, though that is not guaranteed for sequences in general. key in D == key in D.keys() range(len(L)) iter(D) iter(range(Len(L)) == iter(D.keys()) L.clear D.clear L.copy D.copy lambda key: L[key] D.get The purpose of D.get() is to supply a default instead of raising KeyError when key not in D. The lambda function above does not do that. Turning subscripting into a function is a side-effect of D.get. A generic get function: def get(subscriptable, key, default=None): try: return subscriptable[key] except (IndexError, KeyError): return default lambda: enumerate(L)D.items As I pointed out a couple days ago, an enumerate iterator is quite different from a set-like dynamic view. An actual correspondence: enumerate(L) iter(D.items()) Writing a set-like dynamic view of lists corresponding to D.values() or D.items() would be an interesting project. lambda: range(len(L)) D.keys iter(range(len(L)) iter(D.keys()) Already given above. Iterating indexes is now much rarer than iterating dict keys. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
Jussi Piitulainen :
> operator.itemgetter(*selector)(fields) # ==> ('y', 'y', 'x')
>
> [...]
>
> operator.itemgetter(*selector)(field_dict) # ==> ('y', 'y', 'x')
>
> It's not quite the same but it's close and it works the same for
> strings, lists, dicts, ...
Not quite the same, but nicely found anyway.
Marko
--
https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
Random832 : > So can we discuss how a unified method to get a set of all valid > subscripts (and/or subscript-value pairs) on an object would be a > useful thing to have without getting bogged down in theoretical > claptrap about the meaning of the mapping contract? One could compose a table of correspondences: --- list (L)dict (D) --- L[key] = value D[key] = value del L[key] (*) del L[key] key >= 0 and key < len(L) key in D range(len(L)) iter(D) L.clear D.clear L.copy D.copy lambda key: L[key] D.get lambda: enumerate(L)D.items lambda: range(len(L)) D.keys ...... --- (*) reassigns all keys Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On 31/03/2016 14:27, Random832 wrote: On Thu, Mar 31, 2016, at 09:17, Mark Lawrence via Python-list wrote: On 31/03/2016 14:08, Antoon Pardon wrote: Op 31-03-16 om 13:57 schreef Chris Angelico: Okay. I'll put a slightly different position: Prove that your proposal is worth discussing by actually giving us an example that we can discuss. So far, this thread has had nothing but toy examples (and bogoexamples that prove nothing beyond that the author knows how to mess with Python - fun, but not a strong argument on either side). Give us some real meat to work with, instead of these drips of tantalizing blood. What a strange request. Whether or not something is worth discussing is a personal judgement. So there can be no proof of such a thing. I would say: judge for yourself and act accordingly. Drivel. This is comp.lang.python, where "Practicality beats purity" every time, not comp.theoretical.claptrap. So can we discuss how a unified method to get a set of all valid subscripts (and/or subscript-value pairs) on an object would be a useful thing to have without getting bogged down in theoretical claptrap about the meaning of the mapping contract? We can discuss anything here until the cows come home, but it's a complete waste of time if the powers that be over on python-ideas and/or python-dev don't agree. This was suggested a day or two back but seems to have gone completely over people's heads. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
Chris Angelico : > Or, even more likely and even more Pythonic: > >>>> [fields[i] for i in selector] > ['y', 'y', 'x'] > > As soon as you get past the easy and obvious case of an existing > function, filter and map quickly fall behind comprehensions in utility > and readability. The general need is contexts where you need fields[?] act as a function. Of course, lambda i: fields[i] does it. However, weirdly, dicts have get but lists don't. Ok, dict.get() provides for a default value, but couldn't that come in handy for lists as well? Again, lambda would do it for both dicts and lists: lambda i: fields[i] if i >= 0 and i < len(fields) else default lambda key: fields[key] if key in fields else default Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On Thursday, March 31, 2016 at 6:38:56 PM UTC+5:30, Antoon Pardon wrote: > Op 31-03-16 om 13:57 schreef Chris Angelico: > > Okay. I'll put a slightly different position: Prove that your proposal > > is worth discussing by actually giving us an example that we can > > discuss. So far, this thread has had nothing but toy examples (and > > bogoexamples that prove nothing beyond that the author knows how to > > mess with Python - fun, but not a strong argument on either side). > > Give us some real meat to work with, instead of these drips of > > tantalizing blood. > > What a strange request. Whether or not something is worth discussing > is a personal judgement. So there can be no proof of such a thing. > I would say: judge for yourself and act accordingly. Not been following this thread much And not much interest in the suggestion/request Just thought I'd give a take on what may be the motivation for this There is the allure of One-Fundamental-Data-structure Lisp calls that 'list' [40 years after with more fanfare and less clarity replicated as XML] Math calls that 'function' Even more fundamental in CS than in math That maps are same as functions is standard math. In python one interconverts data→code by going from dict d to d.get code→data by memoization/caching How about a Grand Unified Theory? [Just to be clear -- not my interest or wish] -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On Thu, Mar 31, 2016, at 09:17, Mark Lawrence via Python-list wrote: > On 31/03/2016 14:08, Antoon Pardon wrote: > > Op 31-03-16 om 13:57 schreef Chris Angelico: > >> Okay. I'll put a slightly different position: Prove that your proposal > >> is worth discussing by actually giving us an example that we can > >> discuss. So far, this thread has had nothing but toy examples (and > >> bogoexamples that prove nothing beyond that the author knows how to > >> mess with Python - fun, but not a strong argument on either side). > >> Give us some real meat to work with, instead of these drips of > >> tantalizing blood. > > > > What a strange request. Whether or not something is worth discussing > > is a personal judgement. So there can be no proof of such a thing. > > I would say: judge for yourself and act accordingly. > > Drivel. This is comp.lang.python, where "Practicality beats purity" > every time, not comp.theoretical.claptrap. So can we discuss how a unified method to get a set of all valid subscripts (and/or subscript-value pairs) on an object would be a useful thing to have without getting bogged down in theoretical claptrap about the meaning of the mapping contract? -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On 31/03/2016 14:08, Antoon Pardon wrote: Op 31-03-16 om 13:57 schreef Chris Angelico: Okay. I'll put a slightly different position: Prove that your proposal is worth discussing by actually giving us an example that we can discuss. So far, this thread has had nothing but toy examples (and bogoexamples that prove nothing beyond that the author knows how to mess with Python - fun, but not a strong argument on either side). Give us some real meat to work with, instead of these drips of tantalizing blood. What a strange request. Whether or not something is worth discussing is a personal judgement. So there can be no proof of such a thing. I would say: judge for yourself and act accordingly. Drivel. This is comp.lang.python, where "Practicality beats purity" every time, not comp.theoretical.claptrap. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On 31/03/2016 13:49, Marco Sulla via Python-list wrote: On 31 March 2016 at 14:30, Mark Lawrence via Python-list wrote: Note that dict also supports __getitem__() and __len__(), but is considered a mapping rather than a sequence because the lookups use arbitrary immutable keys rather than integers. Thank you for confirming for what I already wrote and quoted, but I suppose you missed some of my previous messages, my dear friend. Thanks for misquoting me by deliberately snipping the bit about taking this completely useless discussion offline. Please do not "dear friend" me as I don't take kindly to people who go out of their way to waste time and effort on this list. This just isn't going to happen, so please drop it, or do you not realise when you're fighting a losing battle, and it's time to retreat? There are of course other options, you could join in the effort to produce Python 2.8 or RickedPython. I'm sure that they'd welcome some additional help and your patch for your absolutely awesome suggestion. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On Fri, Apr 1, 2016 at 12:08 AM, Antoon Pardon wrote: > Op 31-03-16 om 13:57 schreef Chris Angelico: >> Okay. I'll put a slightly different position: Prove that your proposal >> is worth discussing by actually giving us an example that we can >> discuss. So far, this thread has had nothing but toy examples (and >> bogoexamples that prove nothing beyond that the author knows how to >> mess with Python - fun, but not a strong argument on either side). >> Give us some real meat to work with, instead of these drips of >> tantalizing blood. > > What a strange request. Whether or not something is worth discussing > is a personal judgement. So there can be no proof of such a thing. > I would say: judge for yourself and act accordingly. Plonk. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
Op 31-03-16 om 13:57 schreef Chris Angelico: > Okay. I'll put a slightly different position: Prove that your proposal > is worth discussing by actually giving us an example that we can > discuss. So far, this thread has had nothing but toy examples (and > bogoexamples that prove nothing beyond that the author knows how to > mess with Python - fun, but not a strong argument on either side). > Give us some real meat to work with, instead of these drips of > tantalizing blood. What a strange request. Whether or not something is worth discussing is a personal judgement. So there can be no proof of such a thing. I would say: judge for yourself and act accordingly. -- Antoon -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
Marko Rauhamaa writes:
> Chris Angelico wrote:
>
>> Okay. I'll put a slightly different position: Prove that your
>> proposal is worth discussing by actually giving us an example that we
>> can discuss.
>
> Sorry for missing most of the arguments here, but if you are talking
> about treating lists as special cases of dicts, I have occasionally
> instinctively wanted something like this:
>
> >>> fields = [ "x", "y", "z" ]
> >>> selector = (1, 1, 0)
> >>> list(map(fields.get, selector))
> Traceback (most recent call last):
> File "", line 1, in
> AttributeError: 'list' object has no attribute 'get'
operator.itemgetter(*selector)(fields) # ==> ('y', 'y', 'x')
> analogously with:
>
> >>> field_dict = { 0: "x", 1: "y", 2: "z" }
> >>> list(map(field_dict.get, selector))
> ['y', 'y', 'x']
operator.itemgetter(*selector)(field_dict) # ==> ('y', 'y', 'x')
It's not quite the same but it's close and it works the same for
strings, lists, dicts, ...
--
https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
I want also to add that we are focusing on sequences, but my proposal is also to make map interface more similar, introducing a vdict type that iterates over values, and this will be for me really more practical. PEP 234 ( http://legacy.python.org/dev/peps/pep-0234/ ) never convinced me. Van Rossum said: > The symmetry between "if x in y" and "for x in y" > suggests that it should iterate over keys. This symmetry has been > observed by many independently and has even been used to "explain" > one using the other. This is because for sequences, "if x in y" > iterates over y comparing the iterated values to x. This argument will never convinced me. It's a lot more practical (as Van Rossum admitted further) to iterate over values by default on maps. Furthermore I see much more symmetry between keys of maps and indexes of sequences, so it's much more natural to make map iteration over values by default, as for sequences. This is why I proposed a vdict. On 31 March 2016 at 14:30, Mark Lawrence via Python-list wrote: > Note that dict also supports > __getitem__() and __len__(), but is considered a mapping rather than a > sequence because the lookups use arbitrary immutable keys rather than > integers. Thank you for confirming for what I already wrote and quoted, but I suppose you missed some of my previous messages, my dear friend. -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On Thu, Mar 31, 2016 at 11:36 PM, Marko Rauhamaa wrote: > Chris Angelico : > >> On Thu, Mar 31, 2016 at 10:22 PM, Antoon Pardon >> wrote: >> Okay. I'll put a slightly different position: Prove that your proposal >> is worth discussing by actually giving us an example that we can >> discuss. > > Sorry for missing most of the arguments here, but if you are talking > about treating lists as special cases of dicts, I have occasionally > instinctively wanted something like this: > > >>> fields = [ "x", "y", "z" ] > >>> selector = (1, 1, 0) > >>> list(map(fields.get, selector)) > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'list' object has no attribute 'get' > > Or course, I could: > > >>> list(map(fields.__getitem__, selector)) > ['y', 'y', 'x'] > > but that would abuse a dunder method. So I will need to: > > >>> list(map(lambda i: fields[i], selector)) > ['y', 'y', 'x'] > > or (most likely): > > >>> new_fields = [] > >>> for i in selector: > ... new_fields.append(fields[i]) > ... > >>> new_fields > ['y', 'y', 'x'] > > > This tiny problem of mine could be remedied by adding a get method to > lists. Or, even more likely and even more Pythonic: >>> [fields[i] for i in selector] ['y', 'y', 'x'] As soon as you get past the easy and obvious case of an existing function, filter and map quickly fall behind comprehensions in utility and readability. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
Chris Angelico :
> On Thu, Mar 31, 2016 at 10:22 PM, Antoon Pardon
> wrote:
> Okay. I'll put a slightly different position: Prove that your proposal
> is worth discussing by actually giving us an example that we can
> discuss.
Sorry for missing most of the arguments here, but if you are talking
about treating lists as special cases of dicts, I have occasionally
instinctively wanted something like this:
>>> fields = [ "x", "y", "z" ]
>>> selector = (1, 1, 0)
>>> list(map(fields.get, selector))
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'list' object has no attribute 'get'
analogously with:
>>> field_dict = { 0: "x", 1: "y", 2: "z" }
>>> list(map(field_dict.get, selector))
['y', 'y', 'x']
Or course, I could:
>>> list(map(fields.__getitem__, selector))
['y', 'y', 'x']
but that would abuse a dunder method. So I will need to:
>>> list(map(lambda i: fields[i], selector))
['y', 'y', 'x']
or (most likely):
>>> new_fields = []
>>> for i in selector:
... new_fields.append(fields[i])
...
>>> new_fields
['y', 'y', 'x']
This tiny problem of mine could be remedied by adding a get method to
lists.
Marko
--
https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On 31/03/2016 12:58, Marco Sulla via Python-list wrote: On 31 March 2016 at 04:40, Steven D'Aprano wrote: Enough of the hypothetical arguments about what one could do or might do. Let's see a concrete example of actual real world code used in production, not a mickey-mouse toy program, where it is desirable that adding or deleting one key will modify the rest of the keys in the mapping. 1. the example was for confuting your assertion that an implementation of sequences as extended classes of maps violate the map contract. 2. I already linked a real-world example previously. Google it and you can find tons of examples like that. On 31 March 2016 at 04:44, Steven D'Aprano wrote: for a, b in zip(spam, eggs): # do some stuff, sometimes assign x[a] or b[a] or who knows what? Does this mean that "lists, dicts and zip" should all support the same interface? I do not understand what you mean with this example. A zip object is not a sequence nor a map. My definition of sequences as "ordered maps with integer keys that start from zero and have no gaps" is perfectly valid as I demonstrated to you, while zip objects have nothing in common with sequences and maps, apart the fact they are all iterables. The definition of sequence is given here https://docs.python.org/3/glossary.html#term-sequence. An iterable which supports efficient element access using integer indices via the __getitem__() special method and defines a __len__() method that returns the length of the sequence. Some built-in sequence types are list, str, tuple, and bytes. Note that dict also supports __getitem__() and __len__(), but is considered a mapping rather than a sequence because the lookups use arbitrary immutable keys rather than integers. The collections.abc.Sequence abstract base class defines a much richer interface that goes beyond just __getitem__() and __len__(), adding count(), index(), __contains__(), and __reversed__(). Types that implement this expanded interface can be registered explicitly using register(). As this is a Python list the above definition clearly takes priority over your definition, so can you please take this discussion offline, thanks. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On 31 March 2016 at 04:40, Steven D'Aprano wrote: > Enough of the hypothetical arguments about what one could do or might do. > Let's see a concrete example of actual real world code used in production, > not a mickey-mouse toy program, where it is desirable that adding or > deleting one key will modify the rest of the keys in the mapping. 1. the example was for confuting your assertion that an implementation of sequences as extended classes of maps violate the map contract. 2. I already linked a real-world example previously. Google it and you can find tons of examples like that. On 31 March 2016 at 04:44, Steven D'Aprano wrote: > for a, b in zip(spam, eggs): > # do some stuff, sometimes assign x[a] or b[a] or who knows what? > > > Does this mean that "lists, dicts and zip" should all support the same > interface? I do not understand what you mean with this example. A zip object is not a sequence nor a map. My definition of sequences as "ordered maps with integer keys that start from zero and have no gaps" is perfectly valid as I demonstrated to you, while zip objects have nothing in common with sequences and maps, apart the fact they are all iterables. -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On Thu, Mar 31, 2016 at 10:22 PM, Antoon Pardon wrote: > Op 31-03-16 om 12:36 schreef Steven D'Aprano: >> On Thu, 31 Mar 2016 06:52 pm, Antoon Pardon wrote: >> >>> it is your burden to argue that problem. >> No it isn't. I don't have to do a thing. All I need to do is sit back and >> wait as this discussion peters off into nothing. The burden isn't on me to >> justify the status quo. The burden is on those who want to make this change >> to justify the change, because if you don't, the status quo stays exactly >> the same. > > Sure that is all very well if you stay out of the discussion, or limit > your contribution to mentioning that in your opinion that this is a > very low priority. I have no problem with that. But if you begin to > argue that the proposal has flaws and you argue against it then it > is your intellectual responsibility to support your arguments. > > There is a difference between, (1) this proposal is flawed and (2)I > don't think this is important enough. Starting with the first and then > when pressed to support it, retreating to the second is not fair. > Okay. I'll put a slightly different position: Prove that your proposal is worth discussing by actually giving us an example that we can discuss. So far, this thread has had nothing but toy examples (and bogoexamples that prove nothing beyond that the author knows how to mess with Python - fun, but not a strong argument on either side). Give us some real meat to work with, instead of these drips of tantalizing blood. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
Op 31-03-16 om 12:36 schreef Steven D'Aprano: > On Thu, 31 Mar 2016 06:52 pm, Antoon Pardon wrote: > >> it is your burden to argue that problem. > No it isn't. I don't have to do a thing. All I need to do is sit back and > wait as this discussion peters off into nothing. The burden isn't on me to > justify the status quo. The burden is on those who want to make this change > to justify the change, because if you don't, the status quo stays exactly > the same. Sure that is all very well if you stay out of the discussion, or limit your contribution to mentioning that in your opinion that this is a very low priority. I have no problem with that. But if you begin to argue that the proposal has flaws and you argue against it then it is your intellectual responsibility to support your arguments. There is a difference between, (1) this proposal is flawed and (2)I don't think this is important enough. Starting with the first and then when pressed to support it, retreating to the second is not fair. > And if you're brave enough to take to this Python-Ideas, let alone > Python-Dev, the first question they'll ask is "What's your use-case?". And > since you don't have one, this discussion will go nowhere. So? That doesn't relieve you of your responsibility when you somehow argue that the proposal is flawed, beyond there being no use case. > Oh, there might be a few hundred posts on the topic, from people > bike-shedding it, but unless you convince the core developers, all the > chattering in the world won't get you one nanometre closer to changing the > behaviour of lists and dicts. I have no interest in convincing the core developers. That doesn't mean I can't respond here to arguments against a proposal that are IMO bogus. A bogus argument against a proposal doesn't become a good argument against because there is no use case. > So, Antoon, no, I don't have to justify a single thing. If you want this > change, you have to justify why it should be done. Indeed you don't have to argue against any proposal. You can just sit back and ask for use cases and ask do be convinced this proposal is important enough for the core developers to invest time in. But once you do argue against a proposal, as you did here, you have loaded yourself with the burden to support your argument. Because then your position is that even with an import use case, this is still a bad idea. And if you can't support that position it isn't fair IMO to then retreat to the "no use case" position as if that somehow is a defence of your argued position. -- Antoon. -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
Op 31-03-16 om 12:36 schreef Steven D'Aprano: > On Thu, 31 Mar 2016 06:52 pm, Antoon Pardon wrote: > >> it is your burden to argue that problem. > No it isn't. I don't have to do a thing. If that is how you think about this, why do you contribute? I completly understand if you are of the opinion that other priorities are more important > All I need to do is sit back and > wait as this discussion peters off into nothing. The burden isn't on me to > justify the status quo. The burden is on those who want to make this change > to justify the change, because if you don't, the status quo stays exactly > the same. > > And if you're brave enough to take to this Python-Ideas, let alone > Python-Dev, the first question they'll ask is "What's your use-case?". And > since you don't have one, this discussion will go nowhere. > > Oh, there might be a few hundred posts on the topic, from people > bike-shedding it, but unless you convince the core developers, all the > chattering in the world won't get you one nanometre closer to changing the > behaviour of lists and dicts. > > So, Antoon, no, I don't have to justify a single thing. If you want this > change, you have to justify why it should be done. > > Good luck with that. > > > > -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On Thu, 31 Mar 2016 06:52 pm, Antoon Pardon wrote: > it is your burden to argue that problem. No it isn't. I don't have to do a thing. All I need to do is sit back and wait as this discussion peters off into nothing. The burden isn't on me to justify the status quo. The burden is on those who want to make this change to justify the change, because if you don't, the status quo stays exactly the same. And if you're brave enough to take to this Python-Ideas, let alone Python-Dev, the first question they'll ask is "What's your use-case?". And since you don't have one, this discussion will go nowhere. Oh, there might be a few hundred posts on the topic, from people bike-shedding it, but unless you convince the core developers, all the chattering in the world won't get you one nanometre closer to changing the behaviour of lists and dicts. So, Antoon, no, I don't have to justify a single thing. If you want this change, you have to justify why it should be done. Good luck with that. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
Op 31-03-16 om 04:44 schreef Steven D'Aprano:
> On Thu, 31 Mar 2016 03:52 am, Random832 wrote:
>
>> Like, these are common patterns:
>>
>> for i, x in enumerate(l):
>># do some stuff, sometimes assign l[i]
>>
>> for k, v in d.items():
>># do some stuff, sometimes assign d[k]
>
> for a, b in zip(spam, eggs):
> # do some stuff, sometimes assign x[a] or b[a] or who knows what?
>
>
> Does this mean that "lists, dicts and zip" should all support the same
> interface?
>
> Not every coincidental and trivial piece of similar code is actually
> related.
But your addition shows that the similarities between lists and dicts
are greater than with zip. In the above two examples the assignment
was specific to elements of the list or dict over which was iterated.
Your x[a] and b[a] have no such relationship with the rest of your
example. This is further illustrated by the fact that your solution
doesn't provide a zip branch.
>> A way to apply that pattern generically to an object which may be either
>> a sequence or a mapping might be nice.
> Nice, and easy.
>
> # Duck-typing version.
> def iterpairs(obj):
> if hasattr(obj, 'items'):
> it = obj.items
> else:
> it = enum(obj)
> yield from it
>
>
> # Type-checking version.
> def iterpairs(obj):
> if isinstance(obj, collections.abc.Mapping):
> it = obj.items
> elif isinstance(obj, collections.abc.Sequence):
> it = enum(obj)
> else:
> raise TypeError('not a sequence or a mapping')
> yield from it
>
>
> Pick which one you prefer, stick it in your own personal toolbox of useful
> utilities functions, and off you go.
>
>
>
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
Op 31-03-16 om 04:40 schreef Steven D'Aprano: > On Thu, 31 Mar 2016 06:07 am, Antoon Pardon wrote: > >>> Because fundamentally, it doesn't matter whether dicts are surjections or >>> not. They're still many-to-one mappings, and those mappings between keys >>> and values should not change due to the insertion or deletion of >>> unrelated keys. >> Repeating your opion without further arguments doesn't lend any support >> to that opinion. If within the problem space you are working on, such a >> change would make sense as a step to generate the next mapping from the >> previous one, i don't see what the problem is if one could make use of >> this. > > Enough of the hypothetical arguments about what one could do or might do. > Let's see a concrete example of actual real world code used in production, > not a mickey-mouse toy program, where it is desirable that adding or > deleting one key will modify the rest of the keys in the mapping. I am not going to humour your petitio principii. If you want to argue against such a adjustment on the ground of some characteristics mappings should have, you have to make your case yourself and not try to shift the burden onto the person who has his doubt about your argument. If you continously stress the adaption will loose the stability of mappings and so suggest this is somehow a problem it is your burden to argue that problem. > Arguing for the sake of arguing about what somebody might want is not > productive. Let's see some actual concrete use cases where you have a > database or mapping between keys and values, say something like this: Arguing by just calling attention to a specific characteristic but not giving any reason why it would be bad if people were allowed to use a mapping in a way that violated this characteristic, is not productive either. If you try to argue a principle problem with the proposal, you have to support that and not try to shift the attention to there being no use case for the moment. -- Antoon Pardon. -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
On Thursday 31 March 2016 13:45, Paul Rubin wrote: > Steven D'Aprano writes: >> I want to see an actual application where adding a new key to a .^ >> mapping is expected to change the other keys. > directory["mary.roommate"] = "bob" > directory["mary.address"] = None # unknown address > ... > directory["bob.address"] = "132 Elm Street" > > Since Bob and Mary are roommates, they have the same address, so the > application might want to update both addresses once it learns one of > them. I think I've used that application! It was a nightmare... I had to move house seven times before I stopped getting my ex-roomate Stinky Joe's electricity bill sent to me. Nah, only kidding. I've never seen such an application, and I don't believe it exists. By your use of the word "might", I'm guessing that you've made it up. But even if you didn't, it's still a lousy design. Just because Mary and Bob are roommates *now* doesn't mean that they are joined at the hips like Siamese twins and are roommates forever. If Mary relocates, the application shouldn't automatically relocate Bob as well. And even if the application does, for some strange reason, this logic should be built into the *application itself*, not into dict. By all means subclass dict and make your own fancy mapping type that has all sorts of application- specific smarts (or dumbs, as the case may be). Just don't expect it in the standard dict. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Suggestion: make sequence and map interfaces more similar
Steven D'Aprano writes: > I want to see an actual application where adding a new key to a > mapping is expected to change the other keys. directory["mary.roommate"] = "bob" directory["mary.address"] = None # unknown address ... directory["bob.address"] = "132 Elm Street" Since Bob and Mary are roommates, they have the same address, so the application might want to update both addresses once it learns one of them. -- https://mail.python.org/mailman/listinfo/python-list
