What is the difference between persisting a dataframe and a rdd? When I persist my RDD, the UI says it takes 50G or more of memory. When I persist my dataframe, the UI says it takes 9G or less of memory.
Does the dataframe not persist the actual content? Is it better / faster to persist a RDD when doing a lot of filter, mapping, and collecting operations?
