peter-toth commented on PR #40856: URL: https://github.com/apache/spark/pull/40856#issuecomment-1516659349
> @peter-toth can you briefly explain the idea of fixing it? I've updated the PR recently, but the main change is that the CTE accumulator map argument of `buildCTEMap()` changed from `mutable.HashMap.empty[Long, (CTERelationDef, Int)]` to `mutable.SortedMap.empty[Long, (CTERelationDef, Int, mutable.Map[Long, Int])]`. and the new `mutable.Map[Long, Int]` part tracks where the references are pointing to from a CTE. (The old `Int` part tracks the "count of incoming references".) Once we have this extended outer we can correct the "count of incoming references" in `cleanCTEMap()`. We just need to iterate the keys in reverse order (that's why the outer map is now a `SortedMap`) and if we encounter a CTE whose "count of incoming references" is 0 then we decrease the referenced CTE's "count of incoming references". To build the new inner map `buildCTEMap` has a new `outerRefMap` optional argument. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
