Just write all your transformations at the top of the file and then write the mapreduce chain at the end. If you need to use closures that depend on each other then it may be a bit more ugly but still feasible to implement.
- Ron [image: Facebook] <http://www.facebook.com/ron.reiter> [image: Twitter] <http://twitter.com/#!/ronreiter> [image: LinkedIn] <https://il.linkedin.com/in/ronreiter> On Sun, Oct 21, 2018 at 10:20 PM Marko Ristin-Kaufmann < marko.ris...@gmail.com> wrote: > Hi, > What about writing longer map&reduces in pyspark? When I worked with Spark > and Scala, it was easy to script a longer chain of transformations in > Scala. Does anybody know how that works in Python without multiline lambdas? > > Cheers Marko > > > Le dim. 21 oct. 2018 à 18:40, Ron Reiter <ron.rei...@gmail.com> a écrit : > >> Multi-line lambdas may be nice for functional languages, but Python is >> very imperative in nature and prefers clean syntax over conciseness, which >> means it will make it REALLY hard to fit with Python. I personally find >> multi-line lambdas an unreadable abomination. >> >> As for some more concrete reasons on why this should not be a part of >> Python, see here for an example which shows why it would even be hard to >> come up with a plausible syntax for multi-line lambdas: >> >> https://stackoverflow.com/questions/1233448/no-multiline-lambda-in-python-why-not >> >> Guido's words: >> >> "But the complexity of any proposed solution for this puzzle is immense, >> to me: it requires the parser (or more precisely, the lexer) to be able to >> switch back and forth between indent-sensitive and indent-insensitive >> modes, keeping a stack of previous modes and indentation level. Technically >> that can all be solved (there's already a stack of indentation levels that >> could be generalized). But none of that takes away my gut feeling that it >> is all an elaborate Rube Goldberg contraption." >> >> If one would find a syntax which is Pythonic and clean then I am in favor >> of adding it, but I argue it simply does not exist. >> >> - Ron >> >> >> [image: Facebook] <http://www.facebook.com/ron.reiter> [image: Twitter] >> <http://twitter.com/#!/ronreiter> [image: LinkedIn] >> <https://il.linkedin.com/in/ronreiter> >> >> >> On Sun, Oct 21, 2018 at 7:34 PM Anders Hovmöller <bo...@killingar.net> >> wrote: >> >>> >>> Wheres a lambda expression can be passed anonymously to any other >>> function as an argument. >>> >>> *map(lambda x: x**2, array)* >>> >>> vs >>> >>> *def powers2(x)* >>> * x**2* >>> *map(powers2, array)* >>> >>> Coming up with a name here is not needed, as the operation is expressive >>> enough. >>> >>> >>> Sure, but there is a well known convention for such non-names already: >>> meta syntactical variables. In this case the name could be "foo" and all >>> readers will interpret this as "no name". >>> >>> I admit to have wanted lambdas that are just as powerful as normal >>> functions, but often when I want it the code is nicer when a normal >>> function is used. >>> >>> / Anders >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas@python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas@python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
