Re: Spark DataFrame UNPIVOT feature

Given popularity of related SO questions: - https://stackoverflow.com/q/41670103/1560062 - https://stackoverflow.com/q/42465568/1560062 - https://stackoverflow.com/q/41670103/1560062 it is probably more "nobody thought about asking", than "it is not used often". On Wed, 22 Aug 2018

Re: Spark DataFrame UNPIVOT feature

Hi Reynold/Ivan, People familiar with pandas and R dataframes will likely have used the dataframe "melt" idiom, which is the functionality I believe you are referring to: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.melt.html I have had to write this function myself in my own

Re: Spark DataFrame UNPIVOT feature

Probably just because it is not used that often and nobody has submitted a patch for it. I've used pivot probably on average once a week (primarily in spreadsheets), but I've never used unpivot ... On Tue, Aug 21, 2018 at 3:06 PM Ivan Gozali wrote: > Hi there, > > I was looking into why the

Spark DataFrame UNPIVOT feature

Hi there, I was looking into why the UNPIVOT feature isn't implemented, given that Spark already has PIVOT implemented natively in the DataFrame/Dataset API. Came across this JIRA which talks about implementing PIVOT in Spark 1.6, but no mention