zjuwangg commented on issue #8018:
URL:
https://github.com/apache/incubator-gluten/issues/8018#issuecomment-2499600285
> @zjuwangg I think this part is comparatively tricker than other. Do you
think you can start from an individual PR which adds an utility / API to do
resource estimation on query plans (if the idea aligns with your approach, I am
not sure)? Then in subsequent PRs we can adopt this API in Gluten and
whole-stage transformer for remaining work.
@zhztheplayer Got your meaning, I will try in this way.
> Moreover, does the feature target more for batch query scenarios (ETL,
nightly, etc.)? Since I remember changing a Spark resource profile usually
causes rebooting of executors, which will cause larger latency on ad-hoc
queries?
>
> cc @PHILO-HE
Yes, changing spark resource profile will require resource manager to
allocate more executors, which may cause larger latency. In our production
scenario, we more focus on improving ETL stability(avoid failure)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
