Great thanks for adding more relationship info about `Merge` and `Storage`. @Xiangdong !
As : > (now query engine is in the storage engine..) Merge is a submodule of storage engine and query engine. I think `query engine` and `storage engine` is make sense to me. Here only a suggestion that how about use the `Engine` for both `Storage Engine` and `Query Engine` as they are very close each other and they also have the common share `Merge`. i.e.,: -------- Core Core/TsFile Core/Engine Core/WAL --------------- What to do you think? Best, Jincheng ------------------------------------------------------ 1. --Connectors------------- Connectors components include all data integration with external systems, The components are listed as follows: Connectors Connectors/Grafana Connectors/Spark-TsFile Connectors/Spark-IoTDB Connectors/Hive-tsfile Connectors/Hadoop-tsfile 2. --Client------------- Client components involves RPC between client and server, a command-line interface（multi-languages).The components are listed as follows: Client Client/Java Client/Python 3. --Planner------------- Planner components involves the query process in IoTDB, including multiple types of queries, such as raw data query, aggregations, group by time, group by device, and includes many optimized rules.The components are listed as follows: Planner Planner/sql parser Planner/sql optimizer 4. --Core------------- Core components is the storage engine of IoTDB. The components are listed as follows: Core Core/TsFile Core/Engine Core/WAL 5. --Tools------------- Tools components involves many utils of IoTDB, Such as data Sync. The components are listed as follows: Tools Tools/Sync Tools/Other 6. --Authority--------- Authority 7. --Distribution------ Distribution 8. --Example----------- Example 9. --WebSite----------- WebSite 10. --Doc--------------- Doc 11. --Other------------ Other （I'm not sure if we really need the Other component, but there is no harm in adding one） Xiangdong Huang <saint...@gmail.com> 于2020年1月17日周五 上午12:50写道： > Hi, > > Thanks jincheng and jialin. > > Sorry for absence of the discussion because of busy work these days.. > > Some comments: > > > Core > > Core/TsFile > > Core/Merge > > Core/Storage > > Core/WAL > > Not so clearly.. maybe TsFile, WAL, Storage Engine, Query Engine? (now > query engine is in the storage engine..) > Merge is a submodule of storage engine and query engine. > If we pick out the merge module, we also need to pick out the FLUSH, > MODIFICATION, etc.. I think. > > Best, > ----------------------------------- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > Jialin Qiao <qiaojia...@apache.org> 于2020年1月16日周四 下午11:21写道： > > > Hi Jincheng, > > > > Thank you! Looks good :) > > > > jincheng sun <sunjincheng...@gmail.com> 于2020年1月16日周四 下午4:01写道： > > > > > Thanks for your explanation! @Jialin > > > > > > I thought about the composition of our components from the external > > system, > > > the user level, to the IoTDB bottom-level implementation to the > auxiliary > > > tools, and I adjusted a little based on yours. as follows: > > > > > > 1. --Connectors------------- > > > Connectors components include all data integration with external > systems, > > > The components are listed as follows: > > > > > > Connectors > > > Connectors/Grafana > > > Connectors/Spark-TsFile > > > Connectors/Spark-IoTDB > > > Connectors/Hive-tsfile > > > Connectors/Hadoop-tsfile > > > > > > 2. --Client------------- > > > Client components involves RPC between client and server, a > command-line > > > interface（multi-languages).The components are listed as follows: > > > > > > Client > > > Client/Java > > > Client/Python > > > > > > 3. --Planner------------- > > > Planner components involves the query process in IoTDB, including > > multiple > > > types of queries, such as raw data query, aggregations, group by time, > > > group by device, and includes many optimized rules.The components are > > > listed as follows: > > > > > > Planner > > > Planner/sql parsar > > > Planner/sql optimizer > > > > > > 4. --Core------------- > > > Core components is the storage engine of IoTDB. The components are > listed > > > as follows: > > > > > > Core > > > Core/TsFile > > > Core/Merge > > > Core/Storage > > > Core/WAL > > > > > > 5. --Tools------------- > > > Tools components involves many utils of IoTDB, Such as data Sync. The > > > components are listed as follows: > > > Tools > > > Tools/Sync > > > Tools/Other > > > > > > 6. --Authority--------- > > > Authority > > > > > > 7. --Distribution------ > > > Distribution > > > > > > 8. --Example----------- > > > Example > > > > > > 9. --WebSite----------- > > > WebSite > > > > > > 10. --Doc--------------- > > > Doc > > > > > > 11. --Other------------ > > > Other （I'm not sure if we really need the Other component, but there is > > no > > > harm in adding one） > > > > > > This does not change much from your solution. The core is to add a > layer > > of > > > abstraction. > > > > > > Does this make sense to you? Anyway we can continue to improve if > needed! > > > > > > Best, > > > Jincheng > > > > > > Jialin Qiao <qiaojia...@apache.org> 于2020年1月15日周三 下午4:28写道： > > > > > > > Hi, > > > > > > > > WAL has an independent physical layout that has nothing to do with > > > TsFile. > > > > It is used in the writing process and recovering in IoTDB. > > > > It could be merged into Storage components. > > > > > > > > I'm not sure whether I leave out some components, so I add an Other > and > > > > therefore can not give an example... > > > > > > > > Thanks, > > > > > > > > jincheng sun <sunjincheng...@gmail.com> 于2020年1月15日周三 下午4:01写道： > > > > > > > > > Sorry for the late reply, I just came back from team outing. > > > > > > > > > > Thanks for adding more detail information. I have two questions > then: > > > > > > > > > > - Is WAL a write strategy for TsFlie, can WAL be incorporated into > > > other > > > > > components and what is the reason for being a separate component? > > > > > - Can you give an example of what changes belongs to the Other > > > component? > > > > > > > > > > Best, > > > > > Jincheng > > > > > > > > > > > > > > > Jialin Qiao <qiaojia...@apache.org> 于2020年1月9日周四 下午6:39写道： > > > > > > > > > > > Hi, > > > > > > > > > > > > Sure, maybe a bit more... > > > > > > > > > > > > - TsFile : A columnar file format that has flexible schema and > > data > > > > > > management for time series. > > > > > > - Storage : The storage engine of IoTDB, including writing to > > > > memtable > > > > > > (buffer), flushing, system recovery, and memtable management. > > > > > > - Query : The query process in IoTDB, including multiple types > of > > > > > > queries, such as raw data query, aggregations, group by time, > group > > > by > > > > > > device. > > > > > > - Merge : Compaction of data files, including multiple > compaction > > > > > > strategies, such as (1) merging unsequence TsFile with sequence > > > TsFile, > > > > > (2) > > > > > > merging small sequence TsFiles into a large TsFile. > > > > > > - Sync: Synchronize data files from one IoTDB instance to > > another. > > > > > > - Authority: Administration of IoTDB, including users, roles > and > > > > > > privileges. > > > > > > - WAL : Write-ahead-log. > > > > > > - Client: Rpc between client and server, a command-line > interface > > > and > > > > > > client of different languages. > > > > > > - Distribution : The shared-nothing distribution of IoTDB, > > > including > > > > > raft > > > > > > protocol, data partition, and system information management. > > > > > > - Grafana : The connector for using IoTDB as a data source in > > > > Grafana. > > > > > > - Spark-TsFile : A connector that allows users read/write > TsFile > > > > > through > > > > > > SparkSQL (treat TsFile as an external data source for SparkSQL). > > > > > > - Spark-IoTDB : A connector that allows users read/write IoTDB > > > > through > > > > > > SparkSQL (treat IoTDB as an external data source for SparkSQL). > > > > > > - Hive-tsfile : A connector that maps TsFile as a table in Hive > > for > > > > the > > > > > > query. > > > > > > - Hadoop-tsfile : A connector that allows users read/write > TsFile > > > > > through > > > > > > MapReduce. > > > > > > - Tools : Other tools for IoTDB, such as watermark, memory > > > > estimation, > > > > > > JMX, query history visualization. > > > > > > - Example : Examples of different modules, such as JDBC, > Session, > > > > > TsFile. > > > > > > - WebSite : IoTDB official website. > > > > > > - Doc: documents of IoTDB, docs folder under root directory. > > > > > > - Other: Those do not belong to the above components. > > > > > > > > > > > > > > > > > > jincheng sun <sunjincheng...@gmail.com> 于2020年1月9日周四 下午5:28写道： > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > @Xiangdong thank you for helping to clarify my original > > intention. > > > > you > > > > > > are > > > > > > > right, I really want to list some components to let users > choose > > > > which > > > > > > > component the given issue belongs to. > > > > > > > > > > > > > > @Jialin I think approach 4 would be more meaningful for IoTDB. > > > Could > > > > > you > > > > > > > please add the detailed description for each component, such > as, > > > > > describe > > > > > > > the relationship between `tsfile` and > > > > > > > (`tsfile/hive-tsfile/spark-tsfile/Hadoop-tsfile`), and which > > > changes > > > > > > belong > > > > > > > to `Other` component, etc., which can help us better understand > > the > > > > > > reasons > > > > > > > for such classification. On the other hand, we will further > > discuss > > > > > > whether > > > > > > > there is optimization space for this component classification. > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > Best, > > > > > > > Jincheng > > > > > > > > > > > > > > Xiangdong Huang <saint...@gmail.com> 于2020年1月9日周四 下午4:46写道： > > > > > > > > > > > > > > > > No that is not the Apache Way. No one should be in charge > of > > > an > > > > > area > > > > > > > > > > > > > > > > I think Jincheng's mean is, we can list some components to > let > > > > users > > > > > > > > choose which component the given issue belongs to. > > > > > > > > > > > > > > > > As for which components should be listed, we need to have a > > > > > discussion. > > > > > > > > > > > > > > > > > No one should be in charge of an area > > > > > > > > > > > > > > > > Yes everyone can assign an issue to himself/herself. > > > > > > > > > > > > > > > > Best, > > > > > > > > ---------------------------------- > > > > > > > > Xiangdong Huang > > > > > > > > School of Software, Tsinghua University > > > > > > > > > > > > > > > > 黄向东 > > > > > > > > 清华大学 软件学院 > > > > > > > > > > > > > > > > > > > > > > > > jincheng sun <sunjincheng...@gmail.com> 于2020年1月9日周四 > 下午4:38写道： > > > > > > > > > > > > > > > > > > No that is not the Apache Way. No one should be in charge > > of > > > an > > > > > > area. > > > > > > > > > > > > > > > > > > Oh, great thanks for correcting me Justin, maybe it's my > > > language > > > > > > > > > expression problems. I mean some contributors and > committers > > > will > > > > > be > > > > > > > very > > > > > > > > > interested in some components, then spend more time, and be > > > more > > > > > > > familiar > > > > > > > > > with the code. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Jincheng > > > > > > > > > > > > > > > > > > > > > > > > > > > Justin Mclean <jus...@classsoftware.com> 于2020年1月9日周四 > > > 下午4:27写道： > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > When our community grows stronger, each\ module may > have > > > some > > > > > > > > specific > > > > > > > > > > person in charge. > > > > > > > > > > > > > > > > > > > > No that is not the Apache Way. No one should be in charge > > of > > > an > > > > > > area. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Justin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > ————————————————— > > > > > > Jialin Qiao > > > > > > School of Software, Tsinghua University > > > > > > > > > > > > 乔嘉林 > > > > > > 清华大学 软件学院 > > > > > > > > > > > > > > > > > > > > > > > -- > > > > ————————————————— > > > > Jialin Qiao > > > > School of Software, Tsinghua University > > > > > > > > 乔嘉林 > > > > 清华大学 软件学院 > > > > > > > > > > > > > -- > > ————————————————— > > Jialin Qiao > > School of Software, Tsinghua University > > > > 乔嘉林 > > 清华大学 软件学院 > > >
