EnricoMi opened a new pull request, #36150: URL: https://github.com/apache/spark/pull/36150
### What changes were proposed in this pull request? This proposes a Scala implementation of the `melt` operation. ### Why are the changes needed? The Scala Dataset API provides the `pivot` operation, but not its reverse operation `unpivot` or `melt`. The `melt` operation is part of the Pandas API, which is why this method is provided by PySpark Pandas API, implemented purely in Python. [It should be implemented in Scala](https://github.com/apache/spark/pull/26912#pullrequestreview-332975715) to make this operation available to Scala / Java, and to reuse the implementation in PySpark and PySpark Pandas APIs. ### Does this PR introduce _any_ user-facing change? It adds `melt` to the `Dataset` API. ### How was this patch tested? It is tested in the `DatasetSuite`. TODO: - Agree on the Scala and Java Dataset API - [String based](https://github.com/apache/spark/blob/84980ba343edd9d61cdeae3644b1ecac7bb9a3c9/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L1957-L1971) (column name) or [Column based](https://github.com/apache/spark/blob/a97b8c9047a67a3194a386d88777e837023c7642/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L1957-L1971)? - Use var args or Seq? - Python wrapper -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
