[
https://issues.apache.org/jira/browse/CASSANDRA-13475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16237080#comment-16237080
]
Dikang Gu commented on CASSANDRA-13475:
---------------------------------------
[~bdeggleston], I think I merge most of your questions into the quip, here is a
snapshot of it:
==========================================
Apache Cassandra Pluggable Storage Engine
What is a Cassandra pluggable storage engine
A Cassandra pluggable storage engine is the component in Cassandra database
sever that is responsible for managing how data is stored, both in memory and
disk, and performing actual data I/O operations for a database, as well as
enabling certain features sets that target a specific application need.
More concretely, the storage engine will be **responsible** for:
1. Physical Storage: This involves everything from supporting C* data types and
table schema, as well as the format used for storing data to physical disk.
2. Query: storage engine will support point query and range query of data
stored in the database.
3. Memory Caches: The storage engine may implement row cache or block cache,
for query performance optimization.
4. Advanced Data Types: Like list/map/counter, it's up to storage engine
whether to support the advanced data types or not.
5. Index Support: It's up to storage engine whether to support secondary index
of the stored data or not.
The storage engine will **NOT be responsible **for any distributed or network
features, like schema, gossip, replication, streaming, repair, etc. Those
features need to be implemented on top of the storage engine.
Project Goal
* Clear interface of the Pluggable Storage Engine, which means there is clear
boundary on the storage engine, and we can drop in any storage engine
implementation without any change of other components.
* Refactor existing Cassandra code base to follow the pluggable storage engine
architecture.
Timelines/Guidelines
I expect it will be year long effort to refactor existing storage engine to
follow a mature pluggable storage engine API. During the time, we will refactor
the existing storage engine piece by piece, there should be no regression
(performance, reliability or testability) introduced during the process.
(Very high level) Designs
streaming
Current streaming is coupled with storage engine, but it's not necessary. The
StreamSession class could be very general streaming handling framework. My
proposal is that, for the three streaming phases:
1. Connections Initialization: It could be remain unchanged.
2. Stream preparation phase: We abstract the StreamTransferTask and
StreamReceiveTask, each storage engine will implement its own TransferTask and
ReceiveTask, which hide the details about how to buck read/write to the storage
engine.
3. Streaming phase: Each storage engine implement its own StreamReader and
StreamWriter, to read/write data from/into the stream. On the receiving side,
once the streamed message is fully received, the implementation will be
responsible for ingesting the streamed files into the engine, and make it
available for client requests.
repair
For repair, my idea is that we can keep the high level design, that uses Merkle
trees to calculate the difference, and then uses the streaming framework to
streaming the data. To calculate Merkle trees, different storage engine will
have different implementation, a naive way is to sequential scan a token range
to build the Merkel trees, and then stream the inconsistent token range. It
should be doable. But the incremental repair may not be supported by all
storage engines.
keyspace Metadata
Let's say we can config the storage engine per keyspace, under this design, we
can add a storage engine option in the KeyspaceParams which is stored in
KeyspaceMetadata. We can support setting the storage engine during the creation
of the keyspace, in CQL.
Also, we can support the mechanism to be able to overwrite the option per
server. In this case, streaming between different storage engine needs to be
supported.
When we open or initial a keyspace, we will pick the specific storage engine
based on the option in KeyspaceParams.
table metadata
I think we can keep most of the options in the TableParams,
https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/schema/TableParams.java#L36
The storage engine needs to respect the options in the TableParams, and apply
them if possible. For example, if the storage engine is not a LSM tree based
implementation, then it may not need compaction, then it will ignore that
option.
For storage engine specific options, again, like the compaction, we can move
them out of the general params, and allow to load them from some config files.
Metrics
Each storage engine can implement its own JMX/MBeans, then metrics can still be
exposed through JMX.
read path
Each storage engine has its own implementation of the Partition interface, then
we can re-use existing filters to iterator through the storage, and serve read
requests
Write path
Each storage engine has its own encoding/decoding implementation of the
Cassandra's data model. They also implement the apply(PartitionUpdate)
interface, to write the mutation into storage.
Implementations
Components need to be refactored:
1. **Interfaces**: Abstract Keyspace and ColumnFamilyStore classes, and create
interfaces like IKeyspace and IColumnFamilyStore. Also, storage engine needs to
implement the Partition interface, for query.
2. **Column iterator**: It's very specific to SSTable concept, will be moved to
a **CQLStorageEngine** implementation.
3. **Commit log**: We can keep C*'s commit log, it will live outside of storage
engine, so each engine implementation does not need to worry about it.
4. **Compaction**: Compaction should be completely hiden behind the storage
engine. Each storage engine may or may not need compaction, if they do need,
the implementation could be very different. Current compaction logic should be
moved to the **CQLStorageEngine** implementation.
5. **Context/Counter Contex**t: Need to see if it can be generalized, or is it
a CQLEngine specific thing.
6. **Filter**: I think we can re-use most of the filters, like
ClusteringIndexFilter, ClusteringIndexNamesFilter, ClusterIndexSliceFilter,
etc. The storage engine needs to implement the Partition interface, then the
filter can work with the Partitions.
7. **Lifecycle**: Most of the classes under the db.lifecycle package are
coupled together with the sstable concept, they need to be moved to the
**CQLStorageEngine** implementation. For other storage engine, like rocksdb, it
has it's own transaction implementation, so they probably do not need it.
8. **Marshal**: This package is dealing with data types, could be independent
from storage engine implementation.
9. **Monitoring**: This should be a general monitoring solution.
10. **Partitions**: The partition should be an interface, and each storage
engine will have an implementation of the partition, especially the iterators.
Current implementation will be moved to the **CQLStorageEngine**.
11. **Rows**: This is the in memory presentation of the data, should be on top
of different storage engine implementations.
12. **Transform**: similar as the Rows.
13. **View**: I feel current view implementation is very close to current
storage engine, probably will be a **CQLStorageEngine** specific thing.
14. **DHT**: The token range management should be independent from storage
engine implementation.
15. **Read Path**: As I mentioned above, each storage engine needs to implement
the Partition interface, then a lot of read path code can removed unchanged.
Current implementation will be moved to the **CQLStorageEngine**.
16. **Write Path**: Each storage engine implement how to store the mutations.
Current implementation will be moved to the **CQLStorageEngine**.
17. **Key Cache**: Key cache is very specific to current storage engine, will
be moved to **CQLStorageEngine**.
18. **Row Cache**: This can be a general solution, but if storage engine has
it's own cache implemented, then we may not need the row cache on top of it.
19. **Index**: Like SASI, it's highly coupled with storage engine, different
storage engine will have different index solutions. Current implementation will
be moved to the **CQLStorageEngine**.
20. **Metrics**: Different storage engine may expose different metrics.
21. **Streaming/Repair**: Currently, there is a framework to run the
streaming/repairs. In our RocksDBEngine implementation, we try to fit in
current framework. But the implementation details about very different from
current engine. Current implementation will be moved to the
**CQLStorageEngine**.
22. **Schema**: Schema should be unchanged.
23. **Hints**: Hints should be unchanged.
24. **Tools**: A lot of tools need to be reworked.
> First version of pluggable storage engine API.
> ----------------------------------------------
>
> Key: CASSANDRA-13475
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13475
> Project: Cassandra
> Issue Type: Sub-task
> Reporter: Dikang Gu
> Assignee: Dikang Gu
> Priority: Major
>
> In order to support pluggable storage engine, we need to define a unified
> interface/API, which can allow us to plug in different storage engines for
> different requirements.
> Here is a design quip we are currently working on:
> https://quip.com/bhw5ABUCi3co
> In very high level, the storage engine interface should include APIs to:
> 1. Apply update into the engine.
> 2. Query data from the engine.
> 3. Stream data in/out to/from the engine.
> 4. Table operations, like create/drop/truncate a table, etc.
> 5. Various stats about the engine.
> I create this ticket to start the discussions about the interface.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
