Github user markhamstra commented on the issue:
https://github.com/apache/spark/pull/14332
The Scala API docs clearly specify that the most idiomatic usage of Option
is to treat it as a collection or monad and use map, flatMap, filter, or
foreach. Those docs also clearly specify that using Option values via pattern
matching is less idiomatic. There is nothing fundamentally wrong with `map`
applying a function that produces results of type `Unit`, nor is such an
operation some kind of special not-a-transformation that must always be treated
differently from operations that have been blessed as legitimate
transformations. Sure, when you eventually come to the boundaries of purely
functional code and it is time to deal with side effects, then the distinctness
of `Unit` comes into play. But it is a basic FP principle or best practice to
not treat types as specific instead of abstract until you logically must deal
with the specifics -- and that is just not the general case with `map`
functions that produce `Unit`, nor with folding an Option.
Usage of `Unit` consistent with usage of other types in things like `map`
and `fold` can be confusing to the many Spark contributors who aren't
completely at home with idiomatic FP, and that is why, e.g., we chose not to
allow folding of Option within the Spark code. That's not going to change
based on any argument in this PR's comments.
But neither should we shy away from any and all usages of `Unit` producing
functions within `map` -- especially not when that usage is the preferred,
idiomatic usage of Option. The issue that SPARK-16694 should be dealing with
is not a fundamental problem with `map` and `Unit`, but rather with problems
caused by the not completely sound implementation of `view` in Scala. There
are known problems and ugly corner cases associated with `view`, and that's why
I think Scala programmers would be well advised to avoid it when they can, or
to use it with great caution when they must. Alternatively, you can more
safely use something like Josh Sureth's Viewduction work:
https://github.com/jsuereth/viewducers
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
