*Hi Gluten Community,*
I would like to call for a vote on merging Pull Request #11261, which
introduces the "Bolt" backend to the Apache Gluten (Incubating) project.
*PR Link:* https://github.com/apache/incubator-gluten/pull/11261
*Summary of Changes:* This PR adds support for a new execution backend
based on the open source "Bolt" repository (github.com/bytedance/bolt)
which includes:
- Initial integration of the Bolt backend into the Gluten core without
interfering functionality of the existing backends.
- Comprehensive documentation to facilitate the development setup and
smooth transition from other backends.
- Community validated performance results.
*Highlights of "**Bolt**" backend:*
- Performance & Stability
- Generic features like adaptive task parallelism / operator fusion /
JIT and Spark-specific optimization like row-based spilling, shuffle
offload and memory management offload demonstrate significant 22%
performance enhancement over existing gluten backend on the TPC-DS 1T
benchmark.
- Results consistency with Vanilla Spark is verified on 1million+
daily Spark daily jobs within ByteDance. Known differences like
double type
precision, JSON parse and regex search etc. are re-implemented to match
Spark / Java behavior as much as possible with reasonable & deterministic
behaviors.
- Maintenance Commitment:
- Bolt repository is setup to embrace contributions from everyone by
eliminating opaque CI and privileged merge permissions. We
deeply resonate
with the engineering overhead to maintain internal forks against
unpredictable review patterns upstream, so our goal is to provide an
alternative to replace fragmented rebase with high-velocity, unified
development stream. We have observed many examples that have seriously
hindered the growth of the Gluten community, including but not limited to
- Stale performance improvement after 6 months:
https://github.com/facebookincubator/velox/pull/14472
- Stale bug fix ends up in Gluten workaround after 6 months:
https://github.com/facebookincubator/velox/pull/14722
- Stale bugfix after 12 months:
https://github.com/facebookincubator/velox/pull/11771
- Stale feature after 15 months:
https://github.com/facebookincubator/velox/pull/11088
- 2-year review cycle for performance improvement:
https://github.com/facebookincubator/velox/pull/7066
The Bolt team is committed to ensuring that similar issues
will never occur in the Bolt community. To push for fast
iteration, Github
issues/PR reviews/discussions are currently responded to
within 2 business
days.
- After the merge of the Bolt backend, Bolt team is responsible
for setting up a CI pipeline within the Gluten repository to
ensure quality
& coverage. Besides the existing Gluten committer,
*@guhaiyan0221* *@kexianda
*and *@ZacBlanco *are responsible for maintaining the Bolt backend CI
pipeline stability on all supported hardware platforms.
- Build with community:
- Bolt values “Community over Code” spirit and is collaborating with the
community at early design phase on the following features
- Flink acceleration (*discussion
<https://github.com/bytedance/bolt/discussions/20>*)
- Connector interface for data lake formats (*issue
<https://github.com/bytedance/bolt/issues/107>*)
- Native shuffle with Celeborn (*issue
<https://github.com/bytedance/bolt/issues/13>*)
- Tiered memory manager for spilling & hardware affinity
- …
- We believe the community should own the future of this project.
By nominating early active contributors to committer roles, we are
decentralizing leadership and shaping a governance model built by the
people who use it. This evolution is a critical step toward donating the
project to an open-source foundation
*Review Status:* The PR has received significant feedback and contributions
from several community members. It currently has over 20 commits addressing
style, building scripts, and functional refinements.
*Voting:* The vote is open until 2026-02-8 23:59:59 (UTC-8). According to
the Apache voting process, we need at least 3 +1 votes from PMC
members/Committers and more +1 than -1 votes in total.
Please vote as follows:
- *+1:* Approve the merge
- *+0:* No opinion
- *-1:* Disapprove (please provide specific justification)
Thanks to everyone who has contributed to the reviews and development of
this feature!
Best regards,
Guangxin Wang
