wForget commented on issue #7656: URL: https://github.com/apache/incubator-gluten/issues/7656#issuecomment-2834216149
> [@wForget](https://github.com/wForget) Is the issue still there in your side? looks not fixed. Yes, this issue still exists, but we can avoid it by kyuubi spark sql extension (InsertRebalanceBeforeWrite and FinalStageConfigIsolation optimizers). InsertRebalanceBeforeWrite optimized plan like this: ``` QueryPlans | RebalanceByColumn(part columns) | WriteFileExec ``` Then, we disable coalescePartitions for the final stage: ``` set spark.sql.finalStage.adaptive.coalescePartitions.enabled=false; ``` After that, different hive partitions will be distributed in different tasks, and we can avoid OOM caused by one velox task writing too many hive partitions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
