I am with Louis and Paul so far I think. I won't repeat their comments but
I think I can extend the issue by pointing out that this breaks refactoring
for the inner contents of `for`. Previously, if you have:
```
for lesson <- section["lessons"], reduce: 0 do
counter ->
# complex multi-line-thing using the lesson and counter
end
```
I can refactor this into:
```
for lesson <- section["lessons"], reduce: 0 do
counter ->
complex_operation(lesson, counter)
end
def complex_thing(lesson, counter) do
# complex multi-line-thing using the lesson and counter
end
```
And everything just works, as is normal in Elixir code. The proposed
changes would (as far as I can see) break this and that feels very
unexpected and foreign.
I sympathize with the problem space, but so far it's a -1 for me on this
particular proposed improvement.
- Ben
On Thursday, December 16, 2021 at 10:02:49 AM UTC-5 José Valim wrote:
> Note: This proposal contains images and rich text that may not display
> correctly in your email. If so, you can read this proposal in a gist
> <https://gist.github.com/josevalim/5c6735a4b90acc1bafdafec09acabe4f>.
>
> There is prior art in languages like Common Lisp, Haskell, and even in C#
> with LINQ on having very powerful comprehensions as part of the language.
> While Elixir comprehensions are already very expressive, allowing you to
> map, filter, reduce, and collect over multiple enumerables at the same
> time, it is still not capable of expressing other constructs, such as
> map_reduce.
>
> The challenge here is how to continue adding more expressive power to
> comprehensions without making the API feel massive. That's why, 7 years
> after v1.0, only two new options have been added to comprehensions, :uniq
> and :reduce, to a total of 3 (:into, :uniq, and :reduce).
> Imperative loops
>
> I have been on the record a couple times saying that, while many problems
> are more cleanly solved with recursion, there is a category of problems
> that are much more elegant with imperative loops. One of those problems
> have been described in the "nested-data-structures-traversal"
> <https://github.com/josevalim/nested-data-structure-traversal>
> repository, with solutions available in many different languages. Please
> read the problem statement in said repository, as I will assume from now on
> that you are familiar with it.
>
> Personally speaking, the most concise and clear solution is the Python
> one, which I reproduce here:
>
> section_counter = 1lesson_counter = 1
> for section in sections:
> if section["reset_lesson_position"]:
> lesson_counter = 1
>
> section["position"] = section_counter
> section_counter += 1
>
> for lesson in section["lessons"]:
> lesson["position"] = lesson_counter
> lesson_counter += 1
>
> There are many things that make this solution clear:
>
> - Reassignment
> - Mutability
> - Sensitive whitespace
>
> Let's compare it with the Elixir solution I wrote and personally prefer
> <https://github.com/josevalim/nested-data-structure-traversal/blob/master/elixir/map_reduce.exs>.
>
> I am pasting an image below which highlights certain aspects:
>
> [image: Screenshot 2021-12-13 at 10 02 48]
> <https://user-images.githubusercontent.com/9582/145821890-6557ea21-e61f-4813-8c54-53c4ea1a9438.png>
>
> -
>
> Lack of reassignment: in Elixir, we can't reassign variables, we can
> only rebind them. The difference is, when you do var = some_value
> inside a if, for, etc, the value won't "leak" to the outer scope. This
> implies two things in the snippet above:
> 1. We need to use Enum.map_reduce/3 and pass the state in and out
> (highlighted in red)
> 2. When resetting the lesson counter, we need both sides of the
> conditional (hihhlighted in yellow)
> -
>
> Lack of mutability: even though we set the lesson counter inside the
> inner map_reduce, we still need to update the lesson inside the
> session (highlighted in green)
> -
>
> Lack of sensitive whitespace: we have two additional lines with end in
> them (highlighted in blue)
>
> As you can see, do-end blocks add very litte noise to the final solution
> compared to sensitive whitespace. In fact, the only reason I brought it up
> is so we can confidently discard it from the discussion from now on. And
> also because there is zero chance of the language suddenly becoming
> whitespace sensitive.
>
> There is also zero chance of us introducing reassignment and making
> mutability first class in Elixir too. The reason for this is because we all
> agree that, the majority of the time, lack of reassignment and lack of
> mutability are features that make our code more readable and understandable
> in the long term. The snippet above is one of the few examples where we are
> on the wrong end of the trade-offs.
>
> Therefore, how can we move forward?
> Comprehensions
>
> Comprehensions in Elixir have always been a syntax sugar to more complex
> data-structure traversals. Do you want to have the cartesian product
> between all points in x and y? You could write this:
>
> Enum.flat_map(x, fn i ->
> Enum.map(y, fn j -> {i, j} end)end)
>
> Or with a comprehension:
>
> for i <- x, j <- y, do: {i, j}
>
> Or maybe you want to brute force your way into finding Pythagorean Triples?
>
> Enum.flat_map(1..20, fn a ->
> Enum.flat_map(1..20, fn b ->
> 1..20
> |> Enum.filter(fn c -> a*a + b*b == c*c end)
> |> Enum.map(fn c -> {a, b, c} end)
> end)end)
>
> Or with a comprehension:
>
> for a <- 1..20,
> b <- 1..20,
> c <- 1..20,
> a*a + b*b == c*c,
> do: {a, b, c}
>
> There is no question the comprehensions are more concise and clearer, once
> you understand their basic syntax elements (which are, at this point,
> common to many languages).
>
> As mentioned in the introduction, we can express map, filter, reduce, and
> collect inside comprehensions. But how can we represent map_reduce in a
> clear and concise way?
> The :map_reduce option
>
> Since we have :reduce in comprehensions, we could introduce :map_reduce.
> The solution above would look like this:
>
> {sections, _acc} =
> for section <- sections, map_reduce: {1, 1} do
> {section_counter, lesson_counter} ->
> lesson_counter = if section["reset_lesson_position"], do: 1, else:
> lesson_counter
>
> {lessons, lesson_counter} =
> for lesson <- section["lessons"], map_reduce: lesson_counter do
> lesson_counter ->
> {Map.put(lesson, "position", lesson_counter), lesson_counter + 1}
> end
>
> section =
> section
> |> Map.put("lessons", lessons)
> |> Map.put("position", section_counter)
>
> {section, {section_counter + 1, lesson_counter}}
> end
>
> While there is a bit less noise compared to the original solution, the
> reduction of noise mostly happened by the removal of modules names and a
> few tokens, such as fn, (, and ). In terms of implementation, there is
> still a lot of book keeping required to manage the variables. Can we do
> better?
> Introducing :let
>
> Our goal is to declare variables that are automatically looped within the
> comprehension. So let's introduce a new option that does exactly that:
> :let. :let expects one or a tuple of variables that will be reused across
> the comprehension. At the end, :let returns a tuple with the
> comprehension elements and the let variables.
>
> Here is how the solution would look like:
>
> section_counter = 1lesson_counter = 1
> {sections, _} =
> for section <- sections,
> let: {section_counter, lesson_counter} do
> lesson_counter = if section["reset_lesson_position"], do: 1, else:
> lesson_counter
>
> {lessons, lesson_counter} =
> for lesson <- section["lessons"], let: lesson_counter do
> lesson = Map.put(lesson, "position", lesson_counter)
> lesson_counter = lesson_counter + 1
> lesson
> end
>
> section =
> section
> |> Map.put("lessons", lessons)
> |> Map.put("position", section_counter)
>
> section_counter = section_counter + 1
> section
> end
>
> The :let option automatically takes care of passing the variables across
> the comprehension, considerably cutting down the noise, without introducing
> any mutability into the language. At the end, for+:let returns the result
> of the comprehension plus the :let variables wrapped in a tuple.
> Extensions
>
> Here are some extensions to the proposal above. Not all of them might be
> available on the initial implementation.
> Let initialization
>
> You can also initialize the variables within let for convenience:
>
> {sections, _} =
> for section <- sections,
> let: {section_counter = 1, lesson_counter = 1} do
>
> This should be available in the initial implementation.
> :reduce vs :let
>
> With :let, :reduce becomes somewhat redundant. For example,
> Enum.group_by/2 could be written as:
>
> for {k, v} <- Enum.reverse(list), reduce: %{} do
> acc -> Map.update(acc, k, [v], &[v | &1])end
>
> with :let:
>
> {_, acc} =
> for {k, v} <- Enum.reverse(list), let: acc = %{} do
> acc = Map.update(acc, k, [v], &[v | &1])
> end
>
> The difference, however, is that :let returns the collection, while
> :reduce does not. While the Elixir compiler could be smart enough to
> optimize away building the collection in the :let case if we don't use
> it, we may want to keep both :let and :reduce options for clarity. If
> this is the case, I propose to align the syntaxes such that :reduce uses
> the same semantics as :let. The only difference is the return type:
>
> for {k, v} <- Enum.reverse(list), reduce: acc = %{} do
> acc = Map.update(acc, k, [v], &[v | &1])end
>
> This can be done in a backwards compatible fashion.
> after
>
> When you look at our solution to the problem using let, we had to
> introduce temporary variables in order to update our let variables:
>
> {lessons, lesson_counter} =
> for lesson <- section["lessons"], let: lesson_counter do
> lesson = Map.put(lesson, "position", lesson_counter)
> lesson_counter = lesson_counter + 1
> lesson
> end
>
> One extension is to add after to the comprehensions, which are computed
> after the result is returned:
>
> {lessons, lesson_counter} =
> for lesson <- section["lessons"], let: lesson_counter do
> Map.put(lesson, "position", lesson_counter)
> after
> lesson_counter = lesson_counter + 1
> end
>
> This does not need to be part of the initial implementation.
> Summary
>
> Feedback on the proposal and extensions is welcome!
>
--
You received this message because you are subscribed to the Google Groups
"elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elixir-lang-core+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elixir-lang-core/22703d4b-60cb-4b0e-83d2-4a122f9147afn%40googlegroups.com.
