RE: Joining 2 dataframes, getting result as nested list/structure in dataframe

2017-08-24 Thread JG Perrin
Thanks Michael – this is a great article… very helpful From: Michael Armbrust [mailto:mich...@databricks.com] Sent: Wednesday, August 23, 2017 4:33 PM To: JG Perrin Cc: user@spark.apache.org Subject: Re: Joining 2 dataframes, getting result as nested list/structure in dataframe You can create

Re: Joining 2 dataframes, getting result as nested list/structure in dataframe

2017-08-23 Thread Michael Armbrust
You can create a nested struct that contains multiple columns using struct(). Here's a pretty complete guide on working with nested data: https://databricks.com/blog/2017/02/23/working-complex-data-formats-structured-streaming-apache-spark-2-1.html On Wed, Aug 23, 2017 at 2:30 PM, JG Perrin wrot

Joining 2 dataframes, getting result as nested list/structure in dataframe

2017-08-23 Thread JG Perrin
Hi folks, I am trying to join 2 dataframes, but I would like to have the result as a list of rows of the right dataframe (dDf in the example) in a column of the left dataframe (cDf in the example). I made it work with one column, but having issues adding more columns/creating a row(?). Seq