Dandandan commented on issue #15382: URL: https://github.com/apache/datafusion/issues/15382#issuecomment-5550837152
> > The rest of the RecordBatch remains unchanged (i.e. no copy). > > @Dandandan Wouldn't this mean our memory consumption would grow? In this model, we'd keep each record batch alive instead of slicing it apart and allocating a new one. If record batches have backing arrays/buffers that remain referenced, won't they stick around for the duration of the query? > > besides that I think this is a pretty cool idea I think it depends - if the join reduces data, then it could grow and we maybe would need to compact it / detect this case. In other cases (each row has at least one match), not copying it might actually reduce peak memory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
