jackylee-ch commented on issue #6995: URL: https://github.com/apache/incubator-gluten/issues/6995#issuecomment-2307123740
> Does this mean you use partial logic for SA duplicate reading detection? Or you don't use SA at all, SA just impacts you. We are using velox cache and SA in our environment and found driver memory OOM recentlly. After analyze the heap.dump, we found the `SA duplicate reading detection` is the reason that memory is raising continually. > Yes, if some key events(SparkListenerStageCompleted) are discarded, the middle states would not be cleared, this should be a bug. I can take a look. Great, thanks. > Here is the config to disable the duplicate reading: spark.gluten.soft-affinity.duplicateReadingDetect.enabled Yeap, we have already use it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
