FelixYBW commented on issue #7656: URL: https://github.com/apache/incubator-gluten/issues/7656#issuecomment-2569974552
> Sorry, I didn't understand your question, are you asking for number of reducers? This query had a single stage with 23663 tasks, where each task does a union of data from 7 different branches and writes the results to a storage location. Your error msg shows the memory is occupied by ArrowContextInstance, it's used by shuffle and velox to arrow converter in parquet writer. If it's in shuffle you may try sort based shuffle. If it's in parquet write, it may because the arrow batch size is too large, may because too many rows in the batch, or too large data size in each row. You may check the batch size in UI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
