[
https://issues.apache.org/jira/browse/IGNITE-16691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Tkalenko updated IGNITE-16691:
-------------------------------------
Description:
Currently Storage configuration in Ignite 3.0 reuses concepts right from Ignite
2.x like DataRegion.
However there are substantial differences between 2.x and 3.0 versions: when
there is only one option of Storage in 2.x (off-heap page-based storage), 3.0
supports multiple types of storage (right now RocksDB implementing LSM trees
model, refactored page-based storage in the near future).
With that in mind we need to refactor current configuration storage to the most
abstract way, so Storage configuration entity would have only its Type (e.g.
LSM or PAGE_MEMORY) and human-readable Name (in one-to-one relationship with
Type).
Other implementation details like DataRegion should be hidden in polymorphic
configuration.
*Implementation proposal:*
* Each storage defines its own configuration (schema) if needed: for example,
use a new configuration root.
* Instead of specifying the date region name for the table configuration
(*TableConfigurationSchema#dataStorage*), the specific storage configuration
(*DataStorageConfigurationSchema*) will be specified with all the required
fields.
* Adding a new storage will go through adding a new module.
** Adding an implementation of
*StorageEngineFactory#createEngine(ConfigurationRegistry configRegistry, Path
partitionsStoreDir)* through the *java.util.ServiceLoader* mechanism to create
new engines.
** Adding an heir (polymorphic instance) of *DataStorageConfigurationSchema*.
** Adding an implementation of *ConfigurationModule* to define all required
configuration, which is also loaded through *java.util.ServiceLoader* mechanism.
** Adding an implementation of *StorageEngine*, it is important that
*StorageEngine#name* be implemented so that it can be mapped to
*DataStorageConfigurationSchema#name* to determine the desired storage.
*API:*
{code:java}
// Configuration schemas.
@Config
public class TableConfigurationSchema {
@ConfigValue
public DataStorageConfigurationSchema dataStorage;
// Other fields.
}
@PolymorphicConfig
public class DataStorageConfigurationSchema {
@PolymorphicId(hasDefault = true)
public String name = "rocksdb";
}
// Interfaces.
public interface StorageEngineFactory {
StorageEngine createEngine(ConfigurationRegistry clusterConfigRegistry,
Path partitionsStoreDir) throws StorageException;
}
public interface StorageEngine {
String name();
TableStorage createTable(TableConfiguration tableCfg);
// Other methods.
}
// Classes.
public class DataStorageManager implements IgniteComponent {
public @Nullable StorageEngine engine(DataStorageConfiguration config) {
// Some code.
}
// Other methods.
}
{code}
was:
Currently Storage configuration in Ignite 3.0 reuses concepts right from Ignite
2.x like DataRegion.
However there are substantial differences between 2.x and 3.0 versions: when
there is only one option of Storage in 2.x (off-heap page-based storage), 3.0
supports multiple types of storage (right now RocksDB implementing LSM trees
model, refactored page-based storage in the near future).
With that in mind we need to refactor current configuration storage to the most
abstract way, so Storage configuration entity would have only its Type (e.g.
LSM or PAGE_MEMORY) and human-readable Name (in one-to-one relationship with
Type).
Other implementation details like DataRegion should be hidden in polymorphic
configuration.
*Implementation proposal:*
* Each storage defines its own configuration (schema) if needed: for example,
use a new configuration root.
* Instead of specifying the date region name for the table configuration
(*TableConfigurationSchema#dataStorage*), the specific storage configuration
(*DataStorageConfigurationSchema*) will be specified with all the required
fields.
* Adding a new storage will go through adding a new module.
** Adding an implementation of
*StorageEngineFactory#createEngine(ConfigurationRegistry configRegistry, Path
partitionsStoreDir)* through the *java.util.ServiceLoader* mechanism to create
new engines.
** Adding an heir (polymorphic instance) of *DataStorageConfigurationSchema*.
** Adding an implementation of *ConfigurationModule* to define all required
configuration, which is also loaded through *java.util.ServiceLoader* mechanism.
** Adding an implementation of *StorageEngine*, it is important that
*StorageEngine#name* be implemented so that it can be mapped to
*DataStorageConfigurationSchema#name* to determine the desired storage.
*API:*
{code:java}
// Configuration schemas.
@Config
public class TableConfigurationSchema {
@ConfigValue
public DataStorageConfigurationSchema dataStorage;
// Other fields.
}
@PolymorphicConfig
public class DataStorageConfigurationSchema {
@PolymorphicId(hasDefault = true)
public String name = "rocksdb";
}
// Interfaces.
public interface StorageEngineFactory {
StorageEngine createEngine(ConfigurationRegistry clusterConfigRegistry,
Path partitionsStoreDir) throws StorageException;
}
public interface StorageEngine {
String name();
TableStorage createTable(TableConfiguration tableCfg);
// Other methods.
}
// Classes.
public class DataStorageManager implements IgniteComponent {
public @Nullable StorageEngine engine(DataStorageConfiguration config) {
// Some code.
}
// Other methods.
}
{code}
> Storage Configuration refactoring and improvements
> --------------------------------------------------
>
> Key: IGNITE-16691
> URL: https://issues.apache.org/jira/browse/IGNITE-16691
> Project: Ignite
> Issue Type: Task
> Reporter: Sergey Chugunov
> Assignee: Kirill Tkalenko
> Priority: Major
> Labels: iep-55, ignite-3
> Fix For: 3.0.0-alpha5
>
>
> Currently Storage configuration in Ignite 3.0 reuses concepts right from
> Ignite 2.x like DataRegion.
> However there are substantial differences between 2.x and 3.0 versions: when
> there is only one option of Storage in 2.x (off-heap page-based storage), 3.0
> supports multiple types of storage (right now RocksDB implementing LSM trees
> model, refactored page-based storage in the near future).
> With that in mind we need to refactor current configuration storage to the
> most abstract way, so Storage configuration entity would have only its Type
> (e.g. LSM or PAGE_MEMORY) and human-readable Name (in one-to-one relationship
> with Type).
> Other implementation details like DataRegion should be hidden in polymorphic
> configuration.
> *Implementation proposal:*
> * Each storage defines its own configuration (schema) if needed: for example,
> use a new configuration root.
> * Instead of specifying the date region name for the table configuration
> (*TableConfigurationSchema#dataStorage*), the specific storage configuration
> (*DataStorageConfigurationSchema*) will be specified with all the required
> fields.
> * Adding a new storage will go through adding a new module.
> ** Adding an implementation of
> *StorageEngineFactory#createEngine(ConfigurationRegistry configRegistry, Path
> partitionsStoreDir)* through the *java.util.ServiceLoader* mechanism to
> create new engines.
> ** Adding an heir (polymorphic instance) of *DataStorageConfigurationSchema*.
> ** Adding an implementation of *ConfigurationModule* to define all required
> configuration, which is also loaded through *java.util.ServiceLoader*
> mechanism.
> ** Adding an implementation of *StorageEngine*, it is important that
> *StorageEngine#name* be implemented so that it can be mapped to
> *DataStorageConfigurationSchema#name* to determine the desired storage.
> *API:*
> {code:java}
> // Configuration schemas.
> @Config
> public class TableConfigurationSchema {
> @ConfigValue
> public DataStorageConfigurationSchema dataStorage;
>
> // Other fields.
> }
> @PolymorphicConfig
> public class DataStorageConfigurationSchema {
> @PolymorphicId(hasDefault = true)
> public String name = "rocksdb";
> }
> // Interfaces.
> public interface StorageEngineFactory {
> StorageEngine createEngine(ConfigurationRegistry clusterConfigRegistry,
> Path partitionsStoreDir) throws StorageException;
> }
> public interface StorageEngine {
> String name();
>
> TableStorage createTable(TableConfiguration tableCfg);
>
> // Other methods.
> }
> // Classes.
> public class DataStorageManager implements IgniteComponent {
> public @Nullable StorageEngine engine(DataStorageConfiguration config) {
> // Some code.
> }
>
> // Other methods.
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
