aocsa commented on pull request #11274: URL: https://github.com/apache/arrow/pull/11274#issuecomment-932528690
> I think this is a good starting point. Although, if I'm using "head", "limit", or a size-limited sink node of any kind, I think my motivation would be concern that I have a lot of data and I would expect the node to limit that. > > For example, a top-k implementation could periodically (when batches_ has some configurable # of rows) run through and discard data. The way it is written now it would still require me to buffer the entire dataset in memory (and/or spillover). > > Admittedly this would be more CPU efficient but I think that can be a reasonable tradeoff in some cases. Do we have a JIRA for a more RAM-efficient top-k sink? L > I think this is a good starting point. Although, if I'm using "head", "limit", or a size-limited sink node of any kind, I think my motivation would be concern that I have a lot of data and I would expect the node to limit that. > > For example, a top-k implementation could periodically (when batches_ has some configurable # of rows) run through and discard data. The way it is written now it would still require me to buffer the entire dataset in memory (and/or spillover). > > Admittedly this would be more CPU efficient but I think that can be a reasonable tradeoff in some cases. Do we have a JIRA for a more RAM-efficient top-k sink? Yes it's here: https://issues.apache.org/jira/browse/ARROW-14202 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
