GitHub user vaijosh created a discussion: [RFC] Cloud Storage Support for Apache HugeGraph
Hi HugeGraph Dev Community, I would like to open a discussion on a proposed architectural improvement: Cloud Storage Support for Apache HugeGraph. **Motivation:** In modern cloud-native environments, compute instances are ephemeral—frequently created and destroyed based on autoscaling policies. HugeGraph currently relies on local disk storage (RocksDB), which creates significant risks for data durability and complicates operational lifecycle management in these dynamic environments. This limitation is consistent with the challenges discussed in [Issue #3079](https://github.com/apache/hugegraph/issues/3079), which highlights the need for more robust storage management. We propose integrating a pluggable cloud-native storage tier to offload data to object storage, effectively decoupling compute from storage to ensure data persistence, reliability, and operational flexibility. **Proposal:** This RFC proposes a new, pluggable storage architecture that shifts HugeGraph from a local-disk-centric model to a cloud-native, storage-first design. In this approach, durable object storage (S3, GCS, ADLS) acts as the primary, authoritative data store, while local disks are used exclusively as a high-performance cache to optimize access for active workloads. By using Raft for consistency and object storage for persistence, we turn local disks into a temporary cache—allowing us to fully decouple compute from our storage layer. **Key Highlights:** - Decoupled Architecture: Separates compute and consensus from durable storage, enabling independent scaling. - Pluggable Framework (SPI): Enables easy integration of new cloud providers without modifying core engine logic. - Operational Resilience: Uses isolated buckets per store for granular control over IAM, billing, and failure domains. - Intelligent Hydration: Supports both pre-hydration at startup and on-demand, read-miss data loading to optimize performance. **Roadmap:** The proposal outlines a three-phase approach: Phase 1 : Core SPI interface and S3 integration. Phase 2 : GCS/ADLS expansion and telemetry/monitoring. Phase 3 : Production hardening, performance optimization, and enterprise documentation. **RFC Document:** You can review the full design and technical specifications here: [RFC: Cloud Storage Support for HugeGraph](https://docs.google.com/document/d/1yzxKX9cpNKqTgYifWc0-zR2otAUeKGba5DSUUzAGSd8/edit) **PoC Implementation:** A working proof-of-concept is available for those who wish to inspect the implementation: [https://github.com/vaijosh/hugegraph/tree/Hstore%2BCloudStorage](https://www.google.com/url?source=gmail&sa=E&q=https://github.com/vaijosh/hugegraph/tree/Hstore%252BCloudStorage) I welcome your feedback, questions, and suggestions on this architectural direction. Best regards, Vaibhav Joshi GitHub link: https://github.com/apache/hugegraph/discussions/3080 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
