yikf commented on PR #8245: URL: https://github.com/apache/incubator-gluten/pull/8245#issuecomment-2550443058
> > AFAIK, the consistent hashing cannot solve the cache miss problem for velox backend, but can bring a small improvement. > > Thank you for helping confirm. Also, is it possible that the [local cache](https://gluten.apache.org/docs/getting-started/localcache/) can be benefited? > > > Yes, especially when using the spark dynamic executor. > > Yes I assume this is another key point. `Dynamic Allocation ON + Local Cache ON`, could this be a typical target scenario of the change? @yikf are you using local cache so far? As @jackylee-ch said, pure soft affinity scheduling using consistent hashing would be better than the current logic in scenarios where executors change, which is the charm of consistent hashing. for local cache, I assume that there will be some benefits from the scheduling optimization as well. We have used it in the TPC-DS, and we are also looking for scenarios in our online environment. Additionally, we would like to use soft affinity scheduling to optimize other scheduling within our organization. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
