decster commented on issue #3382:
URL:
https://github.com/apache/incubator-doris/issues/3382#issuecomment-620362956
I write a roadmap for this project:
M1: basic functionality, create table/read/write pipeline
Scenario: Users can create a simple table, write(insert/update) to table,
query table(could be slow), without persistence(data loss after BE restart).
1. Create table support in FE, add memory storage medium
2. Core: Make Tablet extendable
3. Core: MemTable, MemSubTablet, HashIndex
4. Core: Column, ColumnDelta, Column Reader/Writer
5. Core: Tablet Reader/Writer
6. Write Pipeline: partial row update Kafka record/file format definition,
reader/writer
7. Write Pipeline: extend DeltaWriter and other write pipeline related code
change
8. Metadata new MemTablet support in TabletMeta and meta
9. TableManager support for MemTablet
10. TxManager support for MemTablet
11. Read Pipeline basic ScanNode for MemTablet(full scan only)
12. Core delta compaction implementation
13. Core old version GC implementation
M2 complete core functionality and better performance
Scenario: support multi-column rowkey, string datatype, etc. Support simple
predicate pushdown, resolve potential performance issues.
1. Core: add string/binary datatype support
2. Core: add multi-column RowKey support
3. Core: add delete support
4. Read: ScanNode support simple predicate push down(>, <, =, etc.)
5. Transaction: optimize publish version related code path
6. Performance: resolve potential major performance issues.
M3 data persistence and fault tolerance
Scenario: there will be no data loss after BE restart or BE failure
1. Persistence: incremental snapshot
2. Persistence: Tablet load (from a snapshot and WAL)
3. Persistence: outdated data GC
4. MemTablet copy
5. Old storage engine supports partial row update
6. MemTablet migrate to SSD/HDD
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
