[
https://issues.apache.org/jira/browse/IGNITE-16691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Tkalenko updated IGNITE-16691:
-------------------------------------
Description:
Currently Storage configuration in Ignite 3.0 reuses concepts right from Ignite
2.x like DataRegion.
However there are substantial differences between 2.x and 3.0 versions: when
there is only one option of Storage in 2.x (off-heap page-based storage), 3.0
supports multiple types of storage (right now RocksDB implementing LSM trees
model, refactored page-based storage in the near future).
With that in mind we need to refactor current configuration storage to the most
abstract way, so Storage configuration entity would have only its Type (e.g.
LSM or PAGE_MEMORY) and human-readable Name (in one-to-one relationship with
Type).
Other implementation details like DataRegion should be hidden in polymorphic
configuration.
*Implementation proposal:*
* Each storage defines its own configuration (schema) if needed: for example,
use a new configuration root.
* Instead of specifying the date region name for the table configuration
(*TableConfigurationSchema#dataStorage*), the specific storage configuration
(*DataStorageConfigurationSchema*) will be specified with all the required
fields.
* Adding a new storage will go through adding a new module.
** Adding an implementation of
*StorageEngineFactory#createEngine(ConfigurationRegistry configRegistry, Path
partitionsStoreDir)* through the *java.util.ServiceLoader* mechanism to create
new engines.
** Adding an heir (polymorphic instance) of *DataStorageConfigurationSchema*.
** Adding an implementation of *ConfigurationModule* to define all required
configuration, which is also loaded through *java.util.ServiceLoader* mechanism.
** Adding an implementation of *StorageEngine*, it is important that
*StorageEngine#name* be implemented so that it can be mapped to
*DataStorageConfigurationSchema#name* to determine the desired storage.
*API:*
{code:java}
// Configuration schemas.
@Config
public class TableConfigurationSchema {
@ConfigValue
public DataStorageConfigurationSchema dataStorage;
// Other fields.
}
@PolymorphicConfig
public class DataStorageConfigurationSchema {
@PolymorphicId(hasDefault = true)
public String name = "rocksdb";
}
// Interfaces.
public interface StorageEngineFactory {
StorageEngine createEngine(ConfigurationRegistry clusterConfigRegistry,
Path partitionsStoreDir) throws StorageException;
}
public interface StorageEngine {
String name();
TableStorage createTable(TableConfiguration tableCfg);
// Other methods.
}
// Classes.
public class DataStorageManager implements IgniteComponent {
private final Map<String, StorageEngine> engines;
public @Nullable StorageEngine engine(DataStorageConfiguration config) {
returns engines.get(config.value().name());
}
// Other methods.
}
{code}
*Implementation example for RocksDB (located in ignite-storage-rocksdb):*
{code:java}
@ConfigurationRoot(rootName = RocksDbStorageEngine.ENGINE_NAME, type =
DISTRIBUTED)
public class RocksDbStorageEngineConfigurationSchema {
public static final String DEFAULT_DATA_REGION_NAME = "default";
@Name(DEFAULT_DATA_REGION_NAME)
@ConfigValue
public RocksDbDataRegionConfigurationSchema defaultRegion;
@ExceptKeys(DEFAULT_DATA_REGION_NAME)
@NamedConfigValue
public RocksDbDataRegionConfigurationSchema regions;
}
@PolymorphicConfigInstance(RocksDbStorageEngine.ENGINE_NAME)
public class RocksDbDataStorageConfigurationSchema extends
DataStorageConfigurationSchema {
@Immutable
@Value(hasDefault = true)
public String dataRegion =
RocksDbStorageEngineConfigurationSchema.DEFAULT_DATA_REGION_NAME;
}
@Config
public class RocksDbDataRegionConfigurationSchema {
public static final String ROCKSDB_LRU_CACHE = "lru";
public static final String ROCKSDB_CLOCK_CACHE = "clock";
@InjectedName
public String name;
@Value(hasDefault = true)
public long size = 256 * 1024 * 1024;
@Value(hasDefault = true)
@Min(1)
public long writeBufferSize = 64 * 1024 * 1024;
@OneOf(ROCKSDB_LRU_CACHE)
@Value(hasDefault = true)
public String cache = ROCKSDB_LRU_CACHE;
@Min(-1)
@Value(hasDefault = true)
public int numShardBits = -1;
}
public class RocksDbStorageEngineFactory implements StorageEngineFactory {
@Override
public StorageEngine createEngine(ConfigurationRegistry
clusterConfigRegistry, Path partitionsStoreDir) throws StorageException {
return new
RocksDbStorageEngine(clusterConfigRegistry.getConfiguration(RocksDbStorageEngineConfiguration.KEY),
partitionsStoreDir);
}
}
public class RocksDbStorageEngine implements StorageEngine {
public static final String ENGINE_NAME = "rocksdb";
@Override
public String name() {
return ENGINE_NAME;
}
// Start of all regions and other methods.
}
{code}
was:
Currently Storage configuration in Ignite 3.0 reuses concepts right from Ignite
2.x like DataRegion.
However there are substantial differences between 2.x and 3.0 versions: when
there is only one option of Storage in 2.x (off-heap page-based storage), 3.0
supports multiple types of storage (right now RocksDB implementing LSM trees
model, refactored page-based storage in the near future).
With that in mind we need to refactor current configuration storage to the most
abstract way, so Storage configuration entity would have only its Type (e.g.
LSM or PAGE_MEMORY) and human-readable Name (in one-to-one relationship with
Type).
Other implementation details like DataRegion should be hidden in polymorphic
configuration.
*Implementation proposal:*
* Each storage defines its own configuration (schema) if needed: for example,
use a new configuration root.
* Instead of specifying the date region name for the table configuration
(*TableConfigurationSchema#dataStorage*), the specific storage configuration
(*DataStorageConfigurationSchema*) will be specified with all the required
fields.
* Adding a new storage will go through adding a new module.
** Adding an implementation of
*StorageEngineFactory#createEngine(ConfigurationRegistry configRegistry, Path
partitionsStoreDir)* through the *java.util.ServiceLoader* mechanism to create
new engines.
** Adding an heir (polymorphic instance) of *DataStorageConfigurationSchema*.
** Adding an implementation of *ConfigurationModule* to define all required
configuration, which is also loaded through *java.util.ServiceLoader* mechanism.
** Adding an implementation of *StorageEngine*, it is important that
*StorageEngine#name* be implemented so that it can be mapped to
*DataStorageConfigurationSchema#name* to determine the desired storage.
*API:*
{code:java}
// Configuration schemas.
@Config
public class TableConfigurationSchema {
@ConfigValue
public DataStorageConfigurationSchema dataStorage;
// Other fields.
}
@PolymorphicConfig
public class DataStorageConfigurationSchema {
@PolymorphicId(hasDefault = true)
public String name = "rocksdb";
}
// Interfaces.
public interface StorageEngineFactory {
StorageEngine createEngine(ConfigurationRegistry clusterConfigRegistry,
Path partitionsStoreDir) throws StorageException;
}
public interface StorageEngine {
String name();
TableStorage createTable(TableConfiguration tableCfg);
// Other methods.
}
// Classes.
public class DataStorageManager implements IgniteComponent {
public @Nullable StorageEngine engine(DataStorageConfiguration config) {
// Some code.
}
// Other methods.
}
{code}
> Storage Configuration refactoring and improvements
> --------------------------------------------------
>
> Key: IGNITE-16691
> URL: https://issues.apache.org/jira/browse/IGNITE-16691
> Project: Ignite
> Issue Type: Task
> Reporter: Sergey Chugunov
> Assignee: Kirill Tkalenko
> Priority: Major
> Labels: iep-55, ignite-3
> Fix For: 3.0.0-alpha5
>
>
> Currently Storage configuration in Ignite 3.0 reuses concepts right from
> Ignite 2.x like DataRegion.
> However there are substantial differences between 2.x and 3.0 versions: when
> there is only one option of Storage in 2.x (off-heap page-based storage), 3.0
> supports multiple types of storage (right now RocksDB implementing LSM trees
> model, refactored page-based storage in the near future).
> With that in mind we need to refactor current configuration storage to the
> most abstract way, so Storage configuration entity would have only its Type
> (e.g. LSM or PAGE_MEMORY) and human-readable Name (in one-to-one relationship
> with Type).
> Other implementation details like DataRegion should be hidden in polymorphic
> configuration.
> *Implementation proposal:*
> * Each storage defines its own configuration (schema) if needed: for example,
> use a new configuration root.
> * Instead of specifying the date region name for the table configuration
> (*TableConfigurationSchema#dataStorage*), the specific storage configuration
> (*DataStorageConfigurationSchema*) will be specified with all the required
> fields.
> * Adding a new storage will go through adding a new module.
> ** Adding an implementation of
> *StorageEngineFactory#createEngine(ConfigurationRegistry configRegistry, Path
> partitionsStoreDir)* through the *java.util.ServiceLoader* mechanism to
> create new engines.
> ** Adding an heir (polymorphic instance) of *DataStorageConfigurationSchema*.
> ** Adding an implementation of *ConfigurationModule* to define all required
> configuration, which is also loaded through *java.util.ServiceLoader*
> mechanism.
> ** Adding an implementation of *StorageEngine*, it is important that
> *StorageEngine#name* be implemented so that it can be mapped to
> *DataStorageConfigurationSchema#name* to determine the desired storage.
> *API:*
> {code:java}
> // Configuration schemas.
> @Config
> public class TableConfigurationSchema {
> @ConfigValue
> public DataStorageConfigurationSchema dataStorage;
>
> // Other fields.
> }
> @PolymorphicConfig
> public class DataStorageConfigurationSchema {
> @PolymorphicId(hasDefault = true)
> public String name = "rocksdb";
> }
> // Interfaces.
> public interface StorageEngineFactory {
> StorageEngine createEngine(ConfigurationRegistry clusterConfigRegistry,
> Path partitionsStoreDir) throws StorageException;
> }
> public interface StorageEngine {
> String name();
>
> TableStorage createTable(TableConfiguration tableCfg);
>
> // Other methods.
> }
> // Classes.
> public class DataStorageManager implements IgniteComponent {
> private final Map<String, StorageEngine> engines;
> public @Nullable StorageEngine engine(DataStorageConfiguration config) {
> returns engines.get(config.value().name());
> }
>
> // Other methods.
> }
> {code}
> *Implementation example for RocksDB (located in ignite-storage-rocksdb):*
> {code:java}
> @ConfigurationRoot(rootName = RocksDbStorageEngine.ENGINE_NAME, type =
> DISTRIBUTED)
> public class RocksDbStorageEngineConfigurationSchema {
> public static final String DEFAULT_DATA_REGION_NAME = "default";
> @Name(DEFAULT_DATA_REGION_NAME)
> @ConfigValue
> public RocksDbDataRegionConfigurationSchema defaultRegion;
> @ExceptKeys(DEFAULT_DATA_REGION_NAME)
> @NamedConfigValue
> public RocksDbDataRegionConfigurationSchema regions;
> }
> @PolymorphicConfigInstance(RocksDbStorageEngine.ENGINE_NAME)
> public class RocksDbDataStorageConfigurationSchema extends
> DataStorageConfigurationSchema {
> @Immutable
> @Value(hasDefault = true)
> public String dataRegion =
> RocksDbStorageEngineConfigurationSchema.DEFAULT_DATA_REGION_NAME;
> }
> @Config
> public class RocksDbDataRegionConfigurationSchema {
> public static final String ROCKSDB_LRU_CACHE = "lru";
> public static final String ROCKSDB_CLOCK_CACHE = "clock";
> @InjectedName
> public String name;
>
> @Value(hasDefault = true)
> public long size = 256 * 1024 * 1024;
> @Value(hasDefault = true)
> @Min(1)
> public long writeBufferSize = 64 * 1024 * 1024;
> @OneOf(ROCKSDB_LRU_CACHE)
> @Value(hasDefault = true)
> public String cache = ROCKSDB_LRU_CACHE;
> @Min(-1)
> @Value(hasDefault = true)
> public int numShardBits = -1;
> }
> public class RocksDbStorageEngineFactory implements StorageEngineFactory {
> @Override
> public StorageEngine createEngine(ConfigurationRegistry
> clusterConfigRegistry, Path partitionsStoreDir) throws StorageException {
> return new
> RocksDbStorageEngine(clusterConfigRegistry.getConfiguration(RocksDbStorageEngineConfiguration.KEY),
> partitionsStoreDir);
> }
> }
> public class RocksDbStorageEngine implements StorageEngine {
> public static final String ENGINE_NAME = "rocksdb";
>
> @Override
> public String name() {
> return ENGINE_NAME;
> }
>
> // Start of all regions and other methods.
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
