Not sure what you mean in "its parents have to reuse it by creating new
RDDs".
As SparkPlan.execute returns new RDD every time, you won't expect the cached
RDD can be reused automatically, even you reuse the SparkPlan in several
queries.
Btw, is there any existing ways to reuse SparkPlan?
Internally, in each partition of the resulting RDD[InternalRow], you will
get the same UnsafeRow when iterating the rows. Typical RDD.cache doesn't
work for it. You will get the output with the same rows. Not sure why you
get empty output.
Dataset.cache() is used for caching SQL query results.