Hi Jayesh, thanks found your email quite interesting :)
Regards, Gourav On Wed, Mar 16, 2022 at 8:02 AM Bitfox <bit...@bitfox.top> wrote: > Thank you. that makes sense. > > On Wed, Mar 16, 2022 at 2:03 PM Lalwani, Jayesh <jlalw...@amazon.com> > wrote: > >> The toDF function in scala uses a bit of Scala magic that allows you to >> add methods to existing classes. Here’s a link to explanation >> https://www.oreilly.com/library/view/scala-cookbook/9781449340292/ch01s11.html >> >> >> >> In short, you can implement a class that extends the List class and add >> methods to your list class, and you can implement an implicit converter >> that converts from List to your class. When the Scala compiler sees that >> you are calling a function on a List object that doesn’t exist in the List >> class, it will look for implicit converters that convert List object to >> another object that has the function, and will automatically call it. >> >> So, if you have a class >> >> Class MyList extends List { >> def toDF(colName: String): DataFrame{ >> ….. >> } >> } >> >> and a implicit converter >> implicit def convertListToMyList(list: List): MyList { >> >> …. >> } >> >> when you do >> List("apple","orange","cherry").toDF("fruit") >> >> >> >> Internally, Scala will generate the code as >> convertListToMyList(List("apple","orange","cherry")).toDF("fruit") >> >> >> >> >> >> *From: *Bitfox <bit...@bitfox.top> >> *Date: *Wednesday, March 16, 2022 at 12:06 AM >> *To: *"user @spark" <user@spark.apache.org> >> *Subject: *[EXTERNAL] Question on List to DF >> >> >> >> *CAUTION*: This email originated from outside of the organization. Do >> not click links or open attachments unless you can confirm the sender and >> know the content is safe. >> >> >> >> I am wondering why the list in scala spark can be converted into a >> dataframe directly? >> >> >> >> scala> val df = List("apple","orange","cherry").toDF("fruit") >> >> *df*: *org.apache.spark.sql.DataFrame* = [fruit: string] >> >> >> >> scala> df.show >> >> +------+ >> >> | fruit| >> >> +------+ >> >> | apple| >> >> |orange| >> >> |cherry| >> >> +------+ >> >> >> >> I don't think pyspark can convert that as well. >> >> >> >> Thank you. >> >
