FelixYBW commented on PR #5799: URL: https://github.com/apache/incubator-gluten/pull/5799#issuecomment-2119736174
> > I met similar issue recently, reports as ArrowContext OOM. Is the root cause that the RowVector too larger? or because the data in each row is too large? By default GLuten configure Velox as 4K batch, but looks many Velox operators may exceed 4K limit. > > Does the PR hold on the batch and output part of the batch each time? > > * In my test case, each row is only 20KB in size, but the size of RowVector is very large, which eventually leads to OOM. > * In my test case the number of output lines exceeds the 4K limit is because of the Generate Operator. > * This PR is aimed to hold on the batch and output part of the batch each time Can we use the batch size (4K row) by default as the config? We can still add the threshold but if customer doesn't config threshold, let's use 4K row each time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
