zebsme commented on issue #15321: URL: https://github.com/apache/datafusion/issues/15321#issuecomment-2754297963
> > hi [@alamb](https://github.com/alamb) , Thanks for sharing your great example here. Following your approach, I noticed that when using an Arrow buffer with mmap as a field in the Decoder, running the tests produces the following output: > > ``` > > thread 'memory_limit::test_stringview_external_sort' panicked at datafusion/core/tests/memory_limit/mod.rs:468:32: > > Query execution failed: ResourcesExhausted("Failed to allocate additional 6643090 bytes for ExternalSorterMerge[0] with 27430266 bytes already allocated for this reservation - 1410938 bytes remain available for the total pool") > > ``` > > > > > > > > > > > > > > > > > > > > > > > > It seems that the drop method defined in memmap2 isn't being executed: > > ``` > > impl Drop for MmapInner { > > fn drop(&mut self) { > > let (ptr, len, _) = self.as_mmap_params(); > > > > // Any errors during unmapping/closing are ignored as the only way > > // to report them would be through panicking which is highly discouraged > > // in Drop impls, c.f. https://github.com/rust-lang/lang-team/issues/97 > > unsafe { libc::munmap(ptr, len as libc::size_t) }; > > } > > } > > ``` > > > > > > > > > > > > > > > > > > > > > > > > However, after switching to using mmap directly as a field, the related tests passed successfully. > > This error is triggered by DataFusion's internal memory tracking component, instead of thrown by the OS. So likely there is something wrong with the spill logic, could you share the draft code? In this issue, I've been experimenting with replacing StreamWriter/StreamReader with FileWriter/FileReader to evaluate potential performance improvements using mmap. However directly replacement seems to cause some other problems, see #14868. Those errors above are encountered with my own FileWriter and FileReader implementation. So I don't think it's an issue for our current spill implementation :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
