[
https://issues.apache.org/jira/browse/IGNITE-16691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513918#comment-17513918
]
Kirill Tkalenko commented on IGNITE-16691:
------------------------------------------
[~agura] Please take a look.
> Storage Configuration refactoring and improvements
> --------------------------------------------------
>
> Key: IGNITE-16691
> URL: https://issues.apache.org/jira/browse/IGNITE-16691
> Project: Ignite
> Issue Type: Task
> Reporter: Sergey Chugunov
> Assignee: Kirill Tkalenko
> Priority: Major
> Labels: iep-55, ignite-3
> Fix For: 3.0.0-alpha5
>
>
> Currently Storage configuration in Ignite 3.0 reuses concepts right from
> Ignite 2.x like DataRegion.
> However there are substantial differences between 2.x and 3.0 versions: when
> there is only one option of Storage in 2.x (off-heap page-based storage), 3.0
> supports multiple types of storage (right now RocksDB implementing LSM trees
> model, refactored page-based storage in the near future).
> With that in mind we need to refactor current configuration storage to the
> most abstract way, so Storage configuration entity would have only its Type
> (e.g. LSM or PAGE_MEMORY) and human-readable Name (in one-to-one relationship
> with Type).
> Other implementation details like DataRegion should be hidden in polymorphic
> configuration.
> *Implementation proposal:*
> * Each storage defines its own configuration (schema) if needed: for example,
> use a new configuration root.
> * Instead of specifying the date region name for the table configuration
> (*TableConfigurationSchema#dataStorage*), the specific storage configuration
> (*DataStorageConfigurationSchema*) will be specified with all the required
> fields.
> * Adding a new storage will go through adding a new module.
> ** Adding an implementation of
> *StorageEngineFactory#createEngine(ConfigurationRegistry configRegistry, Path
> partitionsStoreDir)* through the *java.util.ServiceLoader* mechanism to
> create new engines.
> ** Adding an heir (polymorphic instance) of *DataStorageConfigurationSchema*.
> ** Adding an implementation of *ConfigurationModule* to define all required
> configuration, which is also loaded through *java.util.ServiceLoader*
> mechanism.
> ** Adding an implementation of *StorageEngine*, it is important that
> *StorageEngine#name* be implemented so that it can be mapped to
> *DataStorageConfigurationSchema#name* to determine the desired storage.
> *API:*
> {code:java}
> // Configuration schemas.
> @Config
> public class TableConfigurationSchema {
> @ConfigValue
> public DataStorageConfigurationSchema dataStorage;
>
> // Other fields.
> }
> @PolymorphicConfig
> public class DataStorageConfigurationSchema {
> @PolymorphicId(hasDefault = true)
> public String name = "rocksdb";
> }
> // Interfaces.
> public interface StorageEngineFactory {
> StorageEngine createEngine(ConfigurationRegistry clusterConfigRegistry,
> Path partitionsStoreDir) throws StorageException;
> }
> public interface StorageEngine {
> String name();
>
> TableStorage createTable(TableConfiguration tableCfg);
>
> // Other methods.
> }
> // Classes.
> public class DataStorageManager implements IgniteComponent {
> private final Map<String, StorageEngine> engines;
> public @Nullable StorageEngine engine(DataStorageConfiguration config) {
> returns engines.get(config.value().name());
> }
>
> // Other methods.
> }
> {code}
> *Implementation example for RocksDB (located in ignite-storage-rocksdb):*
> {code:java}
> @ConfigurationRoot(rootName = RocksDbStorageEngine.ENGINE_NAME, type =
> DISTRIBUTED)
> public class RocksDbStorageEngineConfigurationSchema {
> public static final String DEFAULT_DATA_REGION_NAME = "default";
> @Name(DEFAULT_DATA_REGION_NAME)
> @ConfigValue
> public RocksDbDataRegionConfigurationSchema defaultRegion;
> @ExceptKeys(DEFAULT_DATA_REGION_NAME)
> @NamedConfigValue
> public RocksDbDataRegionConfigurationSchema regions;
> }
> @PolymorphicConfigInstance(RocksDbStorageEngine.ENGINE_NAME)
> public class RocksDbDataStorageConfigurationSchema extends
> DataStorageConfigurationSchema {
> @Immutable
> @Value(hasDefault = true)
> public String dataRegion =
> RocksDbStorageEngineConfigurationSchema.DEFAULT_DATA_REGION_NAME;
> }
> @Config
> public class RocksDbDataRegionConfigurationSchema {
> public static final String ROCKSDB_LRU_CACHE = "lru";
> public static final String ROCKSDB_CLOCK_CACHE = "clock";
> @InjectedName
> public String name;
>
> @Value(hasDefault = true)
> public long size = 256 * 1024 * 1024;
> @Value(hasDefault = true)
> @Min(1)
> public long writeBufferSize = 64 * 1024 * 1024;
> @OneOf(ROCKSDB_LRU_CACHE)
> @Value(hasDefault = true)
> public String cache = ROCKSDB_LRU_CACHE;
> @Min(-1)
> @Value(hasDefault = true)
> public int numShardBits = -1;
> }
> public class RocksDbStorageEngineFactory implements StorageEngineFactory {
> @Override
> public StorageEngine createEngine(ConfigurationRegistry
> clusterConfigRegistry, Path partitionsStoreDir) throws StorageException {
> return new
> RocksDbStorageEngine(clusterConfigRegistry.getConfiguration(RocksDbStorageEngineConfiguration.KEY),
> partitionsStoreDir);
> }
> }
> public class RocksDbStorageEngine implements StorageEngine {
> public static final String ENGINE_NAME = "rocksdb";
>
> @Override
> public String name() {
> return ENGINE_NAME;
> }
>
> // Start of all regions and other methods.
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
