Heya > Put yourself into the shoes of someone starting with Elixir and having to solve this problem. How many concepts do you have to learn to get this (allegedly simple) task done?
This is the point of view from which I wrote my email- I think this proposal increases the amount of knowledge a newcomer would need to have in order to understand this code. They would both be unable to use their pre-existing understanding of mutable variables (as the rules are very different), and there would be another set of rules to learn that do not exist elsewhere in Elixir. My hunch is that this feature is one for power-users, not one for newcomers. > How many concepts do you have to learn to get this (allegedly simple) task done? Adding a new concept to learn only reduces the number of concepts needed to learn in order to perform a new task if it can replace several of the others. This concept can only be applied within `for`, and then only in a very restricted fashion, so I don't think it would enable new Elixir developers to skip over learning Elixir as it is today. Because of this I believe it's now N+1 concepts to learn. Cheers, Louis On Thu, 16 Dec 2021 at 18:10, José Valim <jose.va...@dashbit.co> wrote: > Hi everyone, > > Thanks for the input so far. I have one favor to ask: could everyone > please think carefully about the problem statement? > > Put yourself into the shoes of someone starting with Elixir and having to > solve this problem. How many concepts do you have to learn to get this > (allegedly simple) task done? I am aware most people in this mailing list > are comfortable with the Enum.map_reduce/3 solution presented, but we need > to be careful to not fall into the trap of "I've climbed this mountain, I > bet others can too!". In the linked repository, there are solutions in many > languages, and Elixir (and most functional programming languages) are worse > than most of the mainstream languages in terms of size, noise, and required > concepts. > > I am NOT saying we need to sacrifice the things that make Elixir be > Elixir. I am NOT proposing to "dumb down" the language. There are valid > points against this proposal! But I think there would be a strong > dissonance if we can't agree on the problem statement and on how noisy and > mechanical the current functional solution looks. :) > > Some comments/replies below (I am using bold for questions/previous > remarks). > > --- > > *> To what extent can one use these mutable values with other language > features?* > > It would only be at the comprehension root. You are correct there would > still cause confusion as people may expect it to apply in other places. > > *> Overall I don't feel this added complexity and potential for confusion > is worth the benefits here. This brings no more expressive power to the > language.* > > The goal was precisely to not add more expressive power (specifically, not > add mutation). :D > > This is a bit unrelated and I can't remember exactly where I read/heard it > but there is a quote along the lines: "Expressive power is a good measure > for language capabilities but it is not a good measure for language > features". Otherwise, we would all be writing fun(arg1)(arg2) instead of > fun(arg1, arg2). :) > > *> It’d be great to have somewhat like `take: 1` option to stop > evaluation immediately after it has been emitted into `do:` clause.* > > There is already an open issue for the :take option. > > *> I won't repeat their comments but I think I can extend the issue by > pointing out that this breaks refactoring for the inner contents of `for`* > > Correct. This is personally my biggest concern against this proposal, > especially because we have removed features in the past (imperative > assignments for conditionals) because they broke this property. > > > On Thu, Dec 16, 2021 at 5:54 PM Ben Wilson <benwilson...@gmail.com> wrote: > >> I am with Louis and Paul so far I think. I won't repeat their comments >> but I think I can extend the issue by pointing out that this breaks >> refactoring for the inner contents of `for`. Previously, if you have: >> >> ``` >> for lesson <- section["lessons"], reduce: 0 do >> counter -> >> # complex multi-line-thing using the lesson and counter >> end >> ``` >> >> I can refactor this into: >> >> ``` >> for lesson <- section["lessons"], reduce: 0 do >> counter -> >> complex_operation(lesson, counter) >> end >> >> def complex_thing(lesson, counter) do >> # complex multi-line-thing using the lesson and counter >> end >> ``` >> >> And everything just works, as is normal in Elixir code. The proposed >> changes would (as far as I can see) break this and that feels very >> unexpected and foreign. >> >> I sympathize with the problem space, but so far it's a -1 for me on this >> particular proposed improvement. >> >> - Ben >> On Thursday, December 16, 2021 at 10:02:49 AM UTC-5 José Valim wrote: >> >>> Note: This proposal contains images and rich text that may not display >>> correctly in your email. If so, you can read this proposal in a gist >>> <https://gist.github.com/josevalim/5c6735a4b90acc1bafdafec09acabe4f>. >>> >>> There is prior art in languages like Common Lisp, Haskell, and even in >>> C# with LINQ on having very powerful comprehensions as part of the >>> language. While Elixir comprehensions are already very expressive, allowing >>> you to map, filter, reduce, and collect over multiple enumerables at >>> the same time, it is still not capable of expressing other constructs, such >>> as map_reduce. >>> >>> The challenge here is how to continue adding more expressive power to >>> comprehensions without making the API feel massive. That's why, 7 years >>> after v1.0, only two new options have been added to comprehensions, >>> :uniq and :reduce, to a total of 3 (:into, :uniq, and :reduce). >>> Imperative loops >>> >>> I have been on the record a couple times saying that, while many >>> problems are more cleanly solved with recursion, there is a category of >>> problems that are much more elegant with imperative loops. One of those >>> problems have been described in the "nested-data-structures-traversal" >>> <https://github.com/josevalim/nested-data-structure-traversal> >>> repository, with solutions available in many different languages. Please >>> read the problem statement in said repository, as I will assume from now on >>> that you are familiar with it. >>> >>> Personally speaking, the most concise and clear solution is the Python >>> one, which I reproduce here: >>> >>> section_counter = 1lesson_counter = 1 >>> for section in sections: >>> if section["reset_lesson_position"]: >>> lesson_counter = 1 >>> >>> section["position"] = section_counter >>> section_counter += 1 >>> >>> for lesson in section["lessons"]: >>> lesson["position"] = lesson_counter >>> lesson_counter += 1 >>> >>> There are many things that make this solution clear: >>> >>> - Reassignment >>> - Mutability >>> - Sensitive whitespace >>> >>> Let's compare it with the Elixir solution I wrote and personally prefer >>> <https://github.com/josevalim/nested-data-structure-traversal/blob/master/elixir/map_reduce.exs>. >>> I am pasting an image below which highlights certain aspects: >>> >>> [image: Screenshot 2021-12-13 at 10 02 48] >>> <https://user-images.githubusercontent.com/9582/145821890-6557ea21-e61f-4813-8c54-53c4ea1a9438.png> >>> >>> - >>> >>> Lack of reassignment: in Elixir, we can't reassign variables, we can >>> only rebind them. The difference is, when you do var = some_value >>> inside a if, for, etc, the value won't "leak" to the outer scope. >>> This implies two things in the snippet above: >>> 1. We need to use Enum.map_reduce/3 and pass the state in and out >>> (highlighted in red) >>> 2. When resetting the lesson counter, we need both sides of the >>> conditional (hihhlighted in yellow) >>> - >>> >>> Lack of mutability: even though we set the lesson counter inside the >>> inner map_reduce, we still need to update the lesson inside the >>> session (highlighted in green) >>> - >>> >>> Lack of sensitive whitespace: we have two additional lines with end >>> in them (highlighted in blue) >>> >>> As you can see, do-end blocks add very litte noise to the final solution >>> compared to sensitive whitespace. In fact, the only reason I brought it up >>> is so we can confidently discard it from the discussion from now on. And >>> also because there is zero chance of the language suddenly becoming >>> whitespace sensitive. >>> >>> There is also zero chance of us introducing reassignment and making >>> mutability first class in Elixir too. The reason for this is because we all >>> agree that, the majority of the time, lack of reassignment and lack of >>> mutability are features that make our code more readable and understandable >>> in the long term. The snippet above is one of the few examples where we are >>> on the wrong end of the trade-offs. >>> >>> Therefore, how can we move forward? >>> Comprehensions >>> >>> Comprehensions in Elixir have always been a syntax sugar to more complex >>> data-structure traversals. Do you want to have the cartesian product >>> between all points in x and y? You could write this: >>> >>> Enum.flat_map(x, fn i -> >>> Enum.map(y, fn j -> {i, j} end)end) >>> >>> Or with a comprehension: >>> >>> for i <- x, j <- y, do: {i, j} >>> >>> Or maybe you want to brute force your way into finding Pythagorean >>> Triples? >>> >>> Enum.flat_map(1..20, fn a -> >>> Enum.flat_map(1..20, fn b -> >>> 1..20 >>> |> Enum.filter(fn c -> a*a + b*b == c*c end) >>> |> Enum.map(fn c -> {a, b, c} end) >>> end)end) >>> >>> Or with a comprehension: >>> >>> for a <- 1..20, >>> b <- 1..20, >>> c <- 1..20, >>> a*a + b*b == c*c, >>> do: {a, b, c} >>> >>> There is no question the comprehensions are more concise and clearer, >>> once you understand their basic syntax elements (which are, at this point, >>> common to many languages). >>> >>> As mentioned in the introduction, we can express map, filter, reduce, >>> and collect inside comprehensions. But how can we represent map_reduce >>> in a clear and concise way? >>> The :map_reduce option >>> >>> Since we have :reduce in comprehensions, we could introduce :map_reduce. >>> The solution above would look like this: >>> >>> {sections, _acc} = >>> for section <- sections, map_reduce: {1, 1} do >>> {section_counter, lesson_counter} -> >>> lesson_counter = if section["reset_lesson_position"], do: 1, else: >>> lesson_counter >>> >>> {lessons, lesson_counter} = >>> for lesson <- section["lessons"], map_reduce: lesson_counter do >>> lesson_counter -> >>> {Map.put(lesson, "position", lesson_counter), lesson_counter + >>> 1} >>> end >>> >>> section = >>> section >>> |> Map.put("lessons", lessons) >>> |> Map.put("position", section_counter) >>> >>> {section, {section_counter + 1, lesson_counter}} >>> end >>> >>> While there is a bit less noise compared to the original solution, the >>> reduction of noise mostly happened by the removal of modules names and a >>> few tokens, such as fn, (, and ). In terms of implementation, there is >>> still a lot of book keeping required to manage the variables. Can we do >>> better? >>> Introducing :let >>> >>> Our goal is to declare variables that are automatically looped within >>> the comprehension. So let's introduce a new option that does exactly that: >>> :let. :let expects one or a tuple of variables that will be reused >>> across the comprehension. At the end, :let returns a tuple with the >>> comprehension elements and the let variables. >>> >>> Here is how the solution would look like: >>> >>> section_counter = 1lesson_counter = 1 >>> {sections, _} = >>> for section <- sections, >>> let: {section_counter, lesson_counter} do >>> lesson_counter = if section["reset_lesson_position"], do: 1, else: >>> lesson_counter >>> >>> {lessons, lesson_counter} = >>> for lesson <- section["lessons"], let: lesson_counter do >>> lesson = Map.put(lesson, "position", lesson_counter) >>> lesson_counter = lesson_counter + 1 >>> lesson >>> end >>> >>> section = >>> section >>> |> Map.put("lessons", lessons) >>> |> Map.put("position", section_counter) >>> >>> section_counter = section_counter + 1 >>> section >>> end >>> >>> The :let option automatically takes care of passing the variables >>> across the comprehension, considerably cutting down the noise, without >>> introducing any mutability into the language. At the end, for+:let >>> returns the result of the comprehension plus the :let variables wrapped >>> in a tuple. >>> Extensions >>> >>> Here are some extensions to the proposal above. Not all of them might be >>> available on the initial implementation. >>> Let initialization >>> >>> You can also initialize the variables within let for convenience: >>> >>> {sections, _} = >>> for section <- sections, >>> let: {section_counter = 1, lesson_counter = 1} do >>> >>> This should be available in the initial implementation. >>> :reduce vs :let >>> >>> With :let, :reduce becomes somewhat redundant. For example, >>> Enum.group_by/2 could be written as: >>> >>> for {k, v} <- Enum.reverse(list), reduce: %{} do >>> acc -> Map.update(acc, k, [v], &[v | &1])end >>> >>> with :let: >>> >>> {_, acc} = >>> for {k, v} <- Enum.reverse(list), let: acc = %{} do >>> acc = Map.update(acc, k, [v], &[v | &1]) >>> end >>> >>> The difference, however, is that :let returns the collection, while >>> :reduce does not. While the Elixir compiler could be smart enough to >>> optimize away building the collection in the :let case if we don't use >>> it, we may want to keep both :let and :reduce options for clarity. If >>> this is the case, I propose to align the syntaxes such that :reduce >>> uses the same semantics as :let. The only difference is the return type: >>> >>> for {k, v} <- Enum.reverse(list), reduce: acc = %{} do >>> acc = Map.update(acc, k, [v], &[v | &1])end >>> >>> This can be done in a backwards compatible fashion. >>> after >>> >>> When you look at our solution to the problem using let, we had to >>> introduce temporary variables in order to update our let variables: >>> >>> {lessons, lesson_counter} = >>> for lesson <- section["lessons"], let: lesson_counter do >>> lesson = Map.put(lesson, "position", lesson_counter) >>> lesson_counter = lesson_counter + 1 >>> lesson >>> end >>> >>> One extension is to add after to the comprehensions, which are computed >>> after the result is returned: >>> >>> {lessons, lesson_counter} = >>> for lesson <- section["lessons"], let: lesson_counter do >>> Map.put(lesson, "position", lesson_counter) >>> after >>> lesson_counter = lesson_counter + 1 >>> end >>> >>> This does not need to be part of the initial implementation. >>> Summary >>> >>> Feedback on the proposal and extensions is welcome! >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "elixir-lang-core" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elixir-lang-core+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elixir-lang-core/22703d4b-60cb-4b0e-83d2-4a122f9147afn%40googlegroups.com >> <https://groups.google.com/d/msgid/elixir-lang-core/22703d4b-60cb-4b0e-83d2-4a122f9147afn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elixir-lang-core+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KOF759bonEQ-5zX3_ebcO7y8ySOqS3_oeDcMdc-6Dwgw%40mail.gmail.com > <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KOF759bonEQ-5zX3_ebcO7y8ySOqS3_oeDcMdc-6Dwgw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CABu8xFCiT_egSe-PmUmoQR%2BmQDmTigvJfF_wKman9_FQKrqYuA%40mail.gmail.com.
