GitHub user frankobe added a comment to the discussion: Add a new backend: Bolt
**1. Brief Introduction** Bolt is a unified acceleration library powering ByteDance’s internal OLAP & ML preprocess use cases including Spark/Flink/Presto/ElasticSearch and quite a few in-house DBMS etc. Initially derived from Velox, Bolt has undergone continuous iterations & tunings to achieve qualitative improvements in both stability and performance. With the focus to embrace open source ecosystems like data lake formats (Paimon/ Hudi / Lance) and results consistency with original engines, Bolt is not guaranteed to be interface compatible with Velox. During the continuous refactoring of Bolt, Gluten is upgraded simultaneously for critical features like adaptive execution and dependence management. Therefore, compatibility cannot be achieved by simply changing the repository (repo) name, and Bolt prefers to become a separate backend. **2. Summary of Changes on the Gluten Side** 1. Shuffle Offload - Morsel Driven: To improve CPU utilization and solve the long-tail problem caused by data skew in tasks, Bolt introduced Morsel Driven support. The open-source version of Shuffle is implemented on the Gluten side, where batches are fetched from Bolt in a single-threaded manner within a single task, which becomes a performance bottleneck. Bolt has improved it by treating Shuffle as an operator in Bolt and offloading it from Gluten to Bolt for parallel processing. 2. Memory Management Offload - In the current open-source version, the usage of off-heap memory is reported to Spark, and Spark performs unified memory management. However, in practical applications, it was found that the average utilization of physical memory is low, and due to the uneven utilization among different stages, reducing the memory configuration through parameter tuning may cause OOM issues. To address this, Bolt has re-implemented the memory management module and offloaded the management of off-heap memory from Spark to Bolt. Bolt can dynamically over-issue memory after reaching the off-heap configuration value based on the actual physical memory usage, thereby reducing the amount of spilled data, improving performance, and reducing the risk of OOM. 3. Operator Fusion - Bolt's RowVector uses columnar storage, but operators such as Aggregation and Window need to convert columnar data to row data before computation. To reduce the cost of row-column conversion, Bolt has implemented CompositeRowVector, which supports mixed row-column storage. For example, in scenarios such as PartialAggregation + Shuffle (RowBased) + FinalAggregation and Sort + Window, one row-column conversion can be reduced (or the split cost of Shuffle can be reduced). Shuffle also supports processing the CompositeRowVector format with mixed row-column storage. <img width="660" height="1808" alt="image" src="https://github.com/user-attachments/assets/4d2f1b05-b60c-41a3-9828-b27a7b9b9f10"; /> <img width="772" height="1280" alt="image" src="https://github.com/user-attachments/assets/b4ec7ae0-03c0-48e9-a1a2-e5dfa53e8a9b"; /> 4. Support for the package management tool Conan 5. Namespace Incompatibility: Bolt uses the namespace bytedance::bolt instead of facebook::velox 6. Other Features: New support for Paimon MOR, etc. GitHub link: https://github.com/apache/incubator-gluten/discussions/10929#discussioncomment-14822826 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
