Hi, I’d like to discuss a proposal to decouple the engine storage layer
from Hazelcast IMap.
Currently, many parts of the engine directly depend on Hazelcast IMap. To
reduce this tight coupling and make it easier to introduce alternative
storage implementations (e.g., RocksDB) in the future, I’m considering
introducing a MapStorage abstraction layer.
The basic idea is:
- Introduce a MapStorage<K, V> interface
- Provide a Hazelcast-based implementation (IMapStorage)
- Use a MapFactory to create MapStorage instances
- Let services depend on MapStorage instead of Hazelcast IMap directly
In addition, for lifecycle management, I’m considering a context-based
approach:
- Introduce a service-level context that holds:
- NodeEngine
- A storage factory for MapStorage
- Initialize this context once during SeaTunnelServer initialization
- Inject this context into services instead of passing NodeEngine
directly
The goal is to further reduce direct coupling between the service layer and
Hazelcast-specific APIs.
Before proceeding with this refactoring, I’d appreciate feedback on:
1. Whether introducing a MapStorage abstraction makes sense in SeaTunnel
Engine.
2. What would be the preferred way to manage its lifecycle.
3. Any concerns about introducing this level of abstraction.
If the direction looks good, I’ll follow up with a concrete implementation
PR.
Thanks!
Best regards, Doyeon (GitHub ID : dybyte)
