How many tables are involved in the SQL join and how do you cache them? If you do unpersist on the DF and run the sdame
Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 15 December 2016 at 22:14, Warren Kim <w...@diablo-technologies.com> wrote: > Playing with TPC-H and comparing performance between cached (serialized > in-memory tables) and uncached (DF from parquet) results in various > SQL queries performing much worse, duration-wise. > > > I see some physical plans have an extra layer of shuffle/sort/merge under > cached scenario. > > > I could do some filtering by key to optimize, but I'm just curious as to > why out-of-the-box planning is more complex and slower when tables are > cached to mem. > > > Thanks! >
