Github user 0x0FFF commented on the pull request:
https://github.com/apache/spark/pull/8658#issuecomment-138851028
@holdenk, there are two ways of implementing this:
1. The one I've done: adding functionality to utilize binary function
operation from `org.apache.spark.sql.functions` to `Column.py` and utilize
`pow()` function from there for `__pow__` and `__rpow__`. Pros: with this
approach you can add other functions from `org.apache.spark.sql.functions` to
`Column.py`. Cons: non-consistent Column APIs between Python and Scala (to be
fair, Scala does not have ** out of the box, so it is a question whether it is
really an inconsistency)
2. Add implementation of `**` and `pow` to `org.apache.spark.sql.Column`
and utilize it in `Column.py` with existing `_bin_op` and `_reverse_op`. Pros:
`**` operation is added to Scala `org.apache.spark.sql.Column` as well, plus it
turns into slightly less new code. Cons: this would cover only `pow()`
implementation, if in the future you would need to utilize other function from
`org.apache.spark.sql.functions` in Python you would have to change
`org.apache.spark.sql.Column` first once again. Plus in Scala the case with
`3**col("a")` will not work as it would look for `**` implementation in `Int`
In my opinion both options are possible, and second one might even be a bit
better. I can easily switch to the second option, I've already implemented it
locally. What is your view?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
