janhoy commented on PR #4910: URL: https://github.com/apache/solr/pull/4910#issuecomment-5678688772
> > Indexes one line at a time, so peak memory depends on the longest line rather than on the request size. > > Our JSON processing is streaming, so we have no "line" length based memory consumption. Hopefully the code here continues that so that we don't needlessly put lines into memory when we can simply parse token by token from the request buffer. This first implementation is ony streaming on a line level, and I think that is a good start, keeping thigs separate and simple. But there seems to be some low hanging fruit that would let Noggit support ND-JSON and gain better memory footprint with low effort, could be explored here or in a follow up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
