I just want to +1 what Louis has said. Overall, this feels like it makes things more complex rather than less, particularly because of the limited scope that the feature applies to, which makes it difficult to fit into one’s intuition as they learn the language.
While the original example is more complex to learn up front for someone new to Elixir and FP, the tools you are learning to use there are applicable to all problems you face, and it builds towards an intuition that will serve you well as you gain experience with the language. This feature feels out of place from that perspective, at least in my opinion. Have you run this by anyone you know that hasn’t really used the language? I’d be very interested to get the opinion of someone that doesn’t have any preconceived notions of what fits or doesn’t in Elixir, and can evaluate what option is easiest to learn. Paul On Thu, Dec 16, 2021, at 1:54 PM, Louis Pilfold wrote: > Heya > > > Put yourself into the shoes of someone starting with Elixir and having to > > solve this problem. How many concepts do you have to learn to get this > > (allegedly simple) task done? > > This is the point of view from which I wrote my email- I think this proposal > increases the amount of knowledge a newcomer would need to have in order to > understand this code. > > They would both be unable to use their pre-existing understanding of mutable > variables (as the rules are very different), and there would be another set > of rules to learn that do not exist elsewhere in Elixir. My hunch is that > this feature is one for power-users, not one for newcomers. > > > How many concepts do you have to learn to get this (allegedly simple) task > > done? > > Adding a new concept to learn only reduces the number of concepts needed to > learn in order to perform a new task if it can replace several of the others. > This concept can only be applied within `for`, and then only in a very > restricted fashion, so I don't think it would enable new Elixir developers to > skip over learning Elixir as it is today. Because of this I believe it's now > N+1 concepts to learn. > > Cheers, > Louis > > On Thu, 16 Dec 2021 at 18:10, José Valim <jose.va...@dashbit.co> wrote: >> Hi everyone, >> >> Thanks for the input so far. I have one favor to ask: could everyone please >> think carefully about the problem statement? >> >> Put yourself into the shoes of someone starting with Elixir and having to >> solve this problem. How many concepts do you have to learn to get this >> (allegedly simple) task done? I am aware most people in this mailing list >> are comfortable with the Enum.map_reduce/3 solution presented, but we need >> to be careful to not fall into the trap of "I've climbed this mountain, I >> bet others can too!". In the linked repository, there are solutions in many >> languages, and Elixir (and most functional programming languages) are worse >> than most of the mainstream languages in terms of size, noise, and required >> concepts. >> >> I am NOT saying we need to sacrifice the things that make Elixir be Elixir. >> I am NOT proposing to "dumb down" the language. There are valid points >> against this proposal! But I think there would be a strong dissonance if we >> can't agree on the problem statement and on how noisy and mechanical the >> current functional solution looks. :) >> >> Some comments/replies below (I am using bold for questions/previous remarks). >> >> --- >> >> *> To what extent can one use these mutable values with other language >> features?* >> >> It would only be at the comprehension root. You are correct there would >> still cause confusion as people may expect it to apply in other places. >> >> *> Overall I don't feel this added complexity and potential for confusion is >> worth the benefits here. This brings no more expressive power to the >> language.* >> >> The goal was precisely to not add more expressive power (specifically, not >> add mutation). :D >> >> This is a bit unrelated and I can't remember exactly where I read/heard it >> but there is a quote along the lines: "Expressive power is a good measure >> for language capabilities but it is not a good measure for language >> features". Otherwise, we would all be writing fun(arg1)(arg2) instead of >> fun(arg1, arg2). :) >> >> *> It’d be great to have somewhat like `take: 1` option to stop evaluation >> immediately after it has been emitted into `do:` clause.* >> >> There is already an open issue for the :take option. >> >> *> I won't repeat their comments but I think I can extend the issue by >> pointing out that this breaks refactoring for the inner contents of `for`* >> >> Correct. This is personally my biggest concern against this proposal, >> especially because we have removed features in the past (imperative >> assignments for conditionals) because they broke this property. >> >> >> On Thu, Dec 16, 2021 at 5:54 PM Ben Wilson <benwilson...@gmail.com> wrote: >>> I am with Louis and Paul so far I think. I won't repeat their comments but >>> I think I can extend the issue by pointing out that this breaks refactoring >>> for the inner contents of `for`. Previously, if you have: >>> >>> ``` >>> for lesson <- section["lessons"], reduce: 0 do >>> counter -> >>> # complex multi-line-thing using the lesson and counter >>> end >>> ``` >>> >>> I can refactor this into: >>> >>> ``` >>> for lesson <- section["lessons"], reduce: 0 do >>> counter -> >>> complex_operation(lesson, counter) >>> end >>> >>> def complex_thing(lesson, counter) do >>> # complex multi-line-thing using the lesson and counter >>> end >>> ``` >>> And everything just works, as is normal in Elixir code. The proposed >>> changes would (as far as I can see) break this and that feels very >>> unexpected and foreign. >>> >>> I sympathize with the problem space, but so far it's a -1 for me on this >>> particular proposed improvement. >>> >>> - Ben >>> On Thursday, December 16, 2021 at 10:02:49 AM UTC-5 José Valim wrote: >>>> Note: This proposal contains images and rich text that may not display >>>> correctly in your email. If so, you can read this proposal in a gist >>>> <https://gist.github.com/josevalim/5c6735a4b90acc1bafdafec09acabe4f>. >>>> >>>> There is prior art in languages like Common Lisp, Haskell, and even in C# >>>> with LINQ on having very powerful comprehensions as part of the language. >>>> While Elixir comprehensions are already very expressive, allowing you to >>>> `map`, `filter`, `reduce`, and `collect` over multiple enumerables at the >>>> same time, it is still not capable of expressing other constructs, such as >>>> `map_reduce`. >>>> >>>> The challenge here is how to continue adding more expressive power to >>>> comprehensions without making the API feel massive. That's why, 7 years >>>> after v1.0, only two new options have been added to comprehensions, >>>> `:uniq` and `:reduce`, to a total of 3 (`:into`, `:uniq`, and `:reduce`). >>>> >>>> Imperative loops >>>> >>>> I have been on the record a couple times saying that, while many problems >>>> are more cleanly solved with recursion, there is a category of problems >>>> that are much more elegant with imperative loops. One of those problems >>>> have been described in the "nested-data-structures-traversal" >>>> <https://github.com/josevalim/nested-data-structure-traversal> repository, >>>> with solutions available in many different languages. Please read the >>>> problem statement in said repository, as I will assume from now on that >>>> you are familiar with it. >>>> >>>> Personally speaking, the most concise and clear solution is the Python >>>> one, which I reproduce here: >>>> >>>> section_counter = 1 >>>> lesson_counter = 1 >>>> >>>> for section in sections: >>>> if section["reset_lesson_position"]: >>>> lesson_counter = 1 >>>> >>>> section["position"] = section_counter >>>> section_counter += 1 >>>> >>>> for lesson in section["lessons"]: >>>> lesson["position"] = lesson_counter >>>> lesson_counter += 1 >>>> There are many things that make this solution clear: >>>> >>>> * Reassignment >>>> * Mutability >>>> * Sensitive whitespace >>>> Let's compare it with the Elixir solution I wrote and personally prefer >>>> <https://github.com/josevalim/nested-data-structure-traversal/blob/master/elixir/map_reduce.exs>. >>>> I am pasting an image below which highlights certain aspects: >>>> >>>> Screenshot 2021-12-13 at 10 02 48 >>>> <https://user-images.githubusercontent.com/9582/145821890-6557ea21-e61f-4813-8c54-53c4ea1a9438.png> >>>> >>>> * Lack of reassignment: in Elixir, we can't reassign variables, we can >>>> only rebind them. The difference is, when you do `var = some_value` inside >>>> a `if`, `for`, etc, the value won't "leak" to the outer scope. This >>>> implies two things in the snippet above: >>>> >>>> 1. We need to use `Enum.map_reduce/3` and pass the state in and out >>>> (highlighted in red) >>>> 2. When resetting the lesson counter, we need both sides of the >>>> conditional (hihhlighted in yellow) >>>> * Lack of mutability: even though we set the lesson counter inside the >>>> inner `map_reduce`, we still need to update the lesson inside the session >>>> (highlighted in green) >>>> >>>> * Lack of sensitive whitespace: we have two additional lines with `end` >>>> in them (highlighted in blue) >>>> >>>> As you can see, do-end blocks add very litte noise to the final solution >>>> compared to sensitive whitespace. In fact, the only reason I brought it up >>>> is so we can confidently discard it from the discussion from now on. And >>>> also because there is zero chance of the language suddenly becoming >>>> whitespace sensitive. >>>> >>>> There is also zero chance of us introducing reassignment and making >>>> mutability first class in Elixir too. The reason for this is because we >>>> all agree that, the majority of the time, lack of reassignment and lack of >>>> mutability are features that make our code more readable and >>>> understandable in the long term. The snippet above is one of the few >>>> examples where we are on the wrong end of the trade-offs. >>>> >>>> Therefore, how can we move forward? >>>> >>>> Comprehensions >>>> >>>> Comprehensions in Elixir have always been a syntax sugar to more complex >>>> data-structure traversals. Do you want to have the cartesian product >>>> between all points in `x` and `y`? You could write this: >>>> >>>> Enum.flat_map(x, fn i -> >>>> Enum.map(y, fn j -> {i, j} end) >>>> end) >>>> Or with a comprehension: >>>> >>>> for i <- x, j <- y, do: {i, j} >>>> Or maybe you want to brute force your way into finding Pythagorean Triples? >>>> >>>> Enum.flat_map(1..20, fn a -> >>>> Enum.flat_map(1..20, fn b -> >>>> 1..20 >>>> |> Enum.filter(fn c -> a*a + b*b == c*c end) >>>> |> Enum.map(fn c -> {a, b, c} end) >>>> end) >>>> end) >>>> Or with a comprehension: >>>> >>>> for a <- 1..20, >>>> b <- 1..20, >>>> c <- 1..20, >>>> a*a + b*b == c*c, >>>> do: {a, b, c} >>>> There is no question the comprehensions are more concise and clearer, once >>>> you understand their basic syntax elements (which are, at this point, >>>> common to many languages). >>>> >>>> As mentioned in the introduction, we can express `map`, `filter`, >>>> `reduce`, and `collect` inside comprehensions. But how can we represent >>>> `map_reduce` in a clear and concise way? >>>> >>>> The `:map_reduce` option >>>> >>>> Since we have `:reduce` in comprehensions, we could introduce >>>> `:map_reduce`. The solution above would look like this: >>>> >>>> {sections, _acc} = >>>> for section <- sections, map_reduce: {1, 1} do >>>> {section_counter, lesson_counter} -> >>>> lesson_counter = if section["reset_lesson_position"], do: 1, else: >>>> lesson_counter >>>> >>>> {lessons, lesson_counter} = >>>> for lesson <- section["lessons"], map_reduce: lesson_counter do >>>> lesson_counter -> >>>> {Map.put(lesson, "position", lesson_counter), lesson_counter + >>>> 1} >>>> end >>>> >>>> section = >>>> section >>>> |> Map.put("lessons", lessons) >>>> |> Map.put("position", section_counter) >>>> >>>> {section, {section_counter + 1, lesson_counter}} >>>> end >>>> While there is a bit less noise compared to the original solution, the >>>> reduction of noise mostly happened by the removal of modules names and a >>>> few tokens, such as `fn`, `(`, and `)`. In terms of implementation, there >>>> is still a lot of book keeping required to manage the variables. Can we do >>>> better? >>>> >>>> Introducing `:let` >>>> >>>> Our goal is to declare variables that are automatically looped within the >>>> comprehension. So let's introduce a new option that does exactly that: >>>> `:let`. `:let` expects one or a tuple of variables that will be reused >>>> across the comprehension. At the end, `:let` returns a tuple with the >>>> comprehension elements and the let variables. >>>> >>>> Here is how the solution would look like: >>>> >>>> section_counter = 1 >>>> lesson_counter = 1 >>>> >>>> {sections, _} = >>>> for section <- sections, >>>> let: {section_counter, lesson_counter} do >>>> lesson_counter = if section["reset_lesson_position"], do: 1, else: >>>> lesson_counter >>>> >>>> {lessons, lesson_counter} = >>>> for lesson <- section["lessons"], let: lesson_counter do >>>> lesson = Map.put(lesson, "position", lesson_counter) >>>> lesson_counter = lesson_counter + 1 >>>> lesson >>>> end >>>> >>>> section = >>>> section >>>> |> Map.put("lessons", lessons) >>>> |> Map.put("position", section_counter) >>>> >>>> section_counter = section_counter + 1 >>>> section >>>> end >>>> The `:let` option automatically takes care of passing the variables across >>>> the comprehension, considerably cutting down the noise, without >>>> introducing any mutability into the language. At the end, `for+:let` >>>> returns the result of the comprehension plus the `:let` variables wrapped >>>> in a tuple. >>>> >>>> Extensions >>>> >>>> Here are some extensions to the proposal above. Not all of them might be >>>> available on the initial implementation. >>>> >>>> Let initialization >>>> >>>> You can also initialize the variables within `let` for convenience: >>>> >>>> {sections, _} = >>>> for section <- sections, >>>> let: {section_counter = 1, lesson_counter = 1} do >>>> >>>> This should be available in the initial implementation. >>>> >>>> `:reduce` vs `:let` >>>> >>>> With `:let`, `:reduce` becomes somewhat redundant. For example, >>>> `Enum.group_by/2` could be written as: >>>> >>>> for {k, v} <- Enum.reverse(list), reduce: %{} do >>>> acc -> Map.update(acc, k, [v], &[v | &1]) >>>> end >>>> with `:let`: >>>> >>>> {_, acc} = >>>> for {k, v} <- Enum.reverse(list), let: acc = %{} do >>>> acc = Map.update(acc, k, [v], &[v | &1]) >>>> end >>>> The difference, however, is that `:let` returns the collection, while >>>> `:reduce` does not. While the Elixir compiler could be smart enough to >>>> optimize away building the collection in the `:let` case if we don't use >>>> it, we may want to keep both `:let` and `:reduce` options for clarity. If >>>> this is the case, I propose to align the syntaxes such that `:reduce` uses >>>> the same semantics as `:let`. The only difference is the return type: >>>> >>>> for {k, v} <- Enum.reverse(list), reduce: acc = %{} do >>>> acc = Map.update(acc, k, [v], &[v | &1]) >>>> end >>>> This can be done in a backwards compatible fashion. >>>> >>>> `after` >>>> >>>> When you look at our solution to the problem using `let`, we had to >>>> introduce temporary variables in order to update our let variables: >>>> >>>> {lessons, lesson_counter} = >>>> for lesson <- section["lessons"], let: lesson_counter do >>>> lesson = Map.put(lesson, "position", lesson_counter) >>>> lesson_counter = lesson_counter + 1 >>>> lesson >>>> end >>>> One extension is to add `after` to the comprehensions, which are computed >>>> after the result is returned: >>>> >>>> {lessons, lesson_counter} = >>>> for lesson <- section["lessons"], let: lesson_counter do >>>> Map.put(lesson, "position", lesson_counter) >>>> after >>>> lesson_counter = lesson_counter + 1 >>>> end >>>> This does not need to be part of the initial implementation. >>>> >>>> Summary >>>> >>>> Feedback on the proposal and extensions is welcome! >>>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "elixir-lang-core" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to elixir-lang-core+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elixir-lang-core/22703d4b-60cb-4b0e-83d2-4a122f9147afn%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/elixir-lang-core/22703d4b-60cb-4b0e-83d2-4a122f9147afn%40googlegroups.com?utm_medium=email&utm_source=footer>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elixir-lang-core" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elixir-lang-core+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KOF759bonEQ-5zX3_ebcO7y8ySOqS3_oeDcMdc-6Dwgw%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KOF759bonEQ-5zX3_ebcO7y8ySOqS3_oeDcMdc-6Dwgw%40mail.gmail.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elixir-lang-core+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elixir-lang-core/CABu8xFCiT_egSe-PmUmoQR%2BmQDmTigvJfF_wKman9_FQKrqYuA%40mail.gmail.com > > <https://groups.google.com/d/msgid/elixir-lang-core/CABu8xFCiT_egSe-PmUmoQR%2BmQDmTigvJfF_wKman9_FQKrqYuA%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/b55240b6-5eb0-4cbc-a1cf-5016a130e1e0%40www.fastmail.com.
