Re: question about catalyst and TreeNode

2016-03-15 Thread Michael Armbrust
Trees are immutable, and TreeNode takes care of copying unchanged parts of the tree when you are doing transformations. As a result, even if you do construct a DAG with the Dataset API, the first transformation will turn it back into a tree. The only exception to this rule is when we share the

question about catalyst and TreeNode

2016-03-15 Thread Koert Kuipers
i am trying to understand some parts of the catalyst optimizer. but i struggle with one bigger picture issue: LogicalPlan extends TreeNode, which makes sense since the optimizations rely on tree transformations like transformUp and transformDown. but how can a LogicalPlan be a tree? isnt it