c21 edited a comment on pull request #35552: URL: https://github.com/apache/spark/pull/35552#issuecomment-1045066962
> I suspect partial agg is not main problem in your internal prod setting as per https://github.com/apache/spark/pull/28804#issuecomment-854089520, but the parallelism of final agg is, correct? @sigmod - I would like to elaborate more about where I come from. I am not pushing back the feature with the goal to work with my internal company environment. In Spark community, we are trying to move forward with [SPIP - storage partitioned join](https://issues.apache.org/jira/browse/SPARK-37375), and deprecate `HashClusteredDistribution` is one part of work. So it's not preferred to add the whole thing back if there's alternative option. For stream-stream join operator, we have to add the distribution `StatefulOpClusteredDistribution` back for state store correctness only, and I don't think it's in the right direction to make it more general. I thought we only wanted to make a special case for SS operators, but look like here we want to keep expanding the functionality. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
