alexeykudinkin commented on PR #5416: URL: https://github.com/apache/hudi/pull/5416#issuecomment-1184818569
> I am not sure why it may lead to a OOM when reader is reading too fast and writing is not able to keep up. Based on my limited knowledge, when consumers consume data quickly and producers produce data relatively slowly, consumers will continue to wait, which affects the throughput of the application. What we want is to allow the producer's data to be delivered to the consumer side as soon as possible, thereby improving CPU usage and throughput. This is why we want to use the queue of Disruptor to improve the efficiency of data flow through a lock-free design. The scenario w/ OOM is the opposite of what you're describing: when producer is faster than the consumer -- in that case, w/o back-pressure mechanism producer will continue to produce filling in the in-memory buffers, with consumer not being able to catch up fast enough, which might eventually end up w/ OOM. So i see some value in having the back-pressure mechanism but that frankly speaking doesn't require us to be grabbing locks in the hot-path -- it could be implemented even w/ Disruptor queue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
