GitHub user wangxinshuo-bolt edited a comment on the discussion: Add a new backend: Bolt
> > Bolt can dynamically over-issue memory after reaching the off-heap > > configuration value based on the actual physical memory usage, thereby > > reducing the amount of spilled data, improving performance, and reducing > > the risk of OOM. > > Looking forward to see this open-sourced — especially how it helps avoid OOM. > We’ve been struggling with that a lot. @boneanxs We have implemented all memory management functions in Bolt, so `Bolt can dynamically over-issue memory after reaching the off-heap configuration value based on the actual physical memory usage`. The specific code is here: https://github.com/bytedance/bolt/blob/c9b1507b696edbae28d5133d6d512a1eac77e446/bolt/common/memory/sparksql/ExecutionMemoryPool.cpp#L347-L419 We will also refactor the memory module later to avoid shortcomings of the existing mechanism. All planning and discussions will be conducted in the bolt repo's issues. GitHub link: https://github.com/apache/incubator-gluten/discussions/10929#discussioncomment-15738427 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
