This is an automated email from the ASF dual-hosted git repository. nju_yaho pushed a commit to branch Ballista-Roadmap in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
commit 79be257707db40d824da2e29b025ea7435128a84 Author: yahoNanJing <[email protected]> AuthorDate: Thu Sep 15 11:45:16 2022 +0800 Update quarterly_roadmap.md --- docs/source/contributor-guide/quarterly_roadmap.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/source/contributor-guide/quarterly_roadmap.md b/docs/source/contributor-guide/quarterly_roadmap.md index c593e859d..954638be1 100644 --- a/docs/source/contributor-guide/quarterly_roadmap.md +++ b/docs/source/contributor-guide/quarterly_roadmap.md @@ -49,6 +49,8 @@ A quarterly roadmap will be published to give the DataFusion community visibilit - Make production ready - Shuffle file cleanup + - Periodically + - Add grpc interface for clients to actively call the cleanup for a job or the whole system - Fill functional gaps between DataFusion and Ballista - Improve task scheduling and data exchange efficiency - Better error handling @@ -57,6 +59,13 @@ A quarterly roadmap will be published to give the DataFusion community visibilit - Schedule restart - Improve monitoring and logging - Auto scaling support + - Better configuration management +- All-at-once job task scheduling +- Shuffle improvement + - Shuffle memory control + - Improve shuffle IO to avoid producing too many files + - Support sort-based shuffle + - Support range partition - Support for multi-scheduler deployments. Initially for resiliency and fault tolerance but ultimately to support sharding for scalability and more efficient caching. - Executor deployment grouping based on resource allocation
