vaijosh opened a new pull request, #3081: URL: https://github.com/apache/hugegraph/pull/3081
# Purpose * **Closes:** #3079 This PR introduces a cloud storage architecture for **HugeGraph Store (HStore)**. In cloud-native environments, storage nodes are ephemeral, making reliance on local disk storage a single point of failure. This implementation decouples the storage layer from local disk dependencies, allowing SST files to be offloaded to durable, scalable cloud storage providers. ## Key Changes ### Pluggable Architecture * Created the `CloudStorageProvider` SPI interface to enable extensible storage backends. * Added `CloudStorageProviderFactory` for seamless provider discovery and lifecycle management. ### S3 Provider Implementation * Introduced the `hg-store-cloud-s3` module, leveraging AWS SDK v2 for production-ready S3/S3 compatible storage interaction. ### Lifecycle & Event Integration * Integrated `CloudStorageEventListener` with `RocksDBFactory`. This hooks into critical SST lifecycle events (`onTableFileCreated`, `onTableFileDeleted`) to ensure synchronized state between local RocksDB and cloud storage. * Implemented startup hydration (`onDBCreated`) to backfill pre-existing files and read-miss on-demand hydration to ensure data availability. ### Infrastructure & Testing * Added a comprehensive local development environment via `docker/cloud-storage/` using MinIO. * Included an integration test script (`test-graph-queries-and-sst.sh`) to verify end-to-end data durability and query consistency. ## Implementation Highlights * **SPI Integration**: Standardized service discovery via `META-INF/services`. * **RocksDB Hooking**: Registered a singleton `RocksdbEventListener` within `RocksDBFactory` to intercept file operations without modifying core RocksDB logic. * **Read-Miss Protection**: Added a guard window mechanism in `onReadMiss` to prevent redundant hydration requests during high-concurrency read scenarios. ## Verifying These Changes * [x] **Unit Tests**: Coverage for config parsing (`CloudStorageConfigTest`), factory registration, and event listener logic. * [x] **Integration Tests**: Verified using the new `docker-compose` setup with MinIO. * [x] **Configuration**: Verified `application.yml` bindings for credentials, bucket management, and sync intervals. ## Impact * **Dependencies**: Added AWS SDK v2; updated `LICENSE` and `NOTICE` accordingly. * **Configurations**: Added `cloud.storage` namespace to `application.yml` (disabled by default). * **Public API**: No breaking changes to existing public-facing client APIs. ## Documentation * Architecture and configuration details are documented in `hugegraph-store/docs/pluggable-cloud-storage-architecture.md`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
