PHILO-HE commented on issue #8018: URL: https://github.com/apache/incubator-gluten/issues/8018#issuecomment-2496676416
@FelixYBW, what we previously considered is how to make resource profile applied for Spark in whole stage fallback situation. We thought it may need some code to hack into Spark code to achieve that. I note this design tried to use a wrapper or a separate configuration. Not sure the feasibility. @zjuwangg, in your design, you mentioned that `WholeStageTransformer` will get resource profile applied for Gluten plans. But `WholeStageTransformer` may only wrap partial offloaded operators for this stage. You have considered this, right? As @Yohahaha mentioned, Spark community has a SPIP for [DynamicExecutorCoreResizing](https://docs.google.com/document/d/1hxEPUirf3eYwNfMOmUHpuI5dIt_HJErCdo7_yr9htQc/edit?pli=1&tab=t.0#heading=h.b6le5qnpmuht) which is another attempt to manage to avoid the OOM failure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
