Re: ElasticByteBufferPool is ever growing thus can cause memory leak.

Hi Everyone, I have committed the initial work of vectored IO in the feature branch. https://github.com/apache/hadoop/commits/feature-vectored-io Created an epic to track the future work https://issues.apache.org/jira/browse/HADOOP-18103. Please try this out and review and let me know what you thi

Re: ElasticByteBufferPool is ever growing thus can cause memory leak.

DirectBufferPool does the weak refs effectively, but it only recycles buffers of the same size; the elastic one will return the first largest buffer. I think we may want to have a variant of the ElasticBufferPool which uses weak refs..we can adopt that and leave the (old) one alone. It is marked a

Re: ElasticByteBufferPool is ever growing thus can cause memory leak.

Thanks for discussing this, Mukund. Another difference between the 2 classes is that ElasticByteBufferPool supports a caller's preference for either on-heap or off-heap buffers and internally calls either allocate or allocateDirect. Do you intend to preserve that functionality too in a single merg

ElasticByteBufferPool is ever growing thus can cause memory leak.

Hello Everyone, I was just browsing through the code while doing my Vectored IO stuff. It seems like ElasticByteBufferPool is an ever growing pool and memory is not getting released as there is no WeakReference being maintained in the pool. This can cause memory leaks in the production environment.