hanahmily opened a new issue, #12913: URL: https://github.com/apache/skywalking/issues/12913
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement. ### Description BanyanDB's query pipeline currently utilizes the iterator pattern for sorting, aggregating, and limiting data. However, in the initial stage of the pipeline—the raw data retrieval—all data in the segments are loaded into memory. This approach can lead to excessive memory usage, especially for heavy aggregation queries, such as retrieving the top 10 items ordered by a tag over a large time range (e.g., "last month"). We propose extending the iterator pattern to the initial raw data retrieval step to address this issue. By doing so, we can significantly reduce memory consumption by streaming data from segments on-demand rather than loading all segment data into memory at once. ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit a pull request to implement this on your own? - [ ] Yes I am willing to submit a pull request on my own! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
