kerwin-zk commented on PR #6671: URL: https://github.com/apache/incubator-gluten/pull/6671#issuecomment-2738918097
> > @marin-ma Since our internal version depends on GLUTEN_RSS_SORT_SHUFFLE_WRITER, I suggest adding a configuration to control whether to use GLUTEN_SORT_SHUFFLE_WRITER or GLUTEN_RSS_SORT_SHUFFLE_WRITER. > > Is your internal version the same as upstream? We picked this up because one customer met OOM issue using upstream. Did you try GLUTEN_SORT_SHUFFLE_WRITER? Is there performance gap? @FelixYBW The internal version is somewhat different from the upstream version, and we haven't encountered any OOM issues so far. I previously tested GLUTEN_SORT_SHUFFLE_WRITER, and there were cases where it failed to process very large datasets. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
