zjuwangg commented on issue #8018: URL: https://github.com/apache/incubator-gluten/issues/8018#issuecomment-2496890426
> @FelixYBW, what we previously considered is how to make resource profile applied for Spark in whole stage fallback situation. We thought it may need some code to hack into Spark code to achieve that. I note this design tried to use a wrapper or a separate configuration. Not sure the feasibility. > > @zjuwangg, in your design, you mentioned that `WholeStageTransformer` will get resource profile applied for Gluten plans. But `WholeStageTransformer` may only wrap partial offloaded operators for this stage. You have considered this, right? > > As @Yohahaha mentioned, Spark community has a SPIP for [DynamicExecutorCoreResizing](https://docs.google.com/document/d/1hxEPUirf3eYwNfMOmUHpuI5dIt_HJErCdo7_yr9htQc/edit?pli=1&tab=t.0#heading=h.b6le5qnpmuht) which is another attempt to manage to avoid the OOM failure. @PHILO-HE > in your design, you mentioned that WholeStageTransformer will get resource profile applied for Gluten plans. But WholeStageTransformer may only wrap partial offloaded operators for this stage. You have considered this, right? There should be no problem, current dynamic resource profile will search underlying rdd's resource profile, once we set the resource profile for WholeStageTransformer, the whole stage will all get affected(whether native node or original spark node.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
