+ 1 jincheng sun <sunjincheng...@gmail.com>于2020年1月24日 周五下午12:33写道：
> I would like to bring up a VOTE mail to make the final decision If there no > objections. > > What do you think? > > Best, > Jincheng > ------------- > Twitter: https://twitter.com/sunjincheng121 > ------------- > > > jincheng sun <sunjincheng...@gmail.com> 于2020年1月17日周五 上午10:39写道： > > > Great thanks for adding more relationship info about `Merge` and > > `Storage`. @Xiangdong ! > > > > As : > > > (now query engine is in the storage engine..) Merge is a submodule of > > storage engine and query engine. > > > > I think `query engine` and `storage engine` is make sense to me. Here > only > > a suggestion that how about use the `Engine` for both `Storage Engine` > and > > `Query Engine` as they are very close each other and they also have the > > common share `Merge`. i.e.,: > > > > -------- > > Core > > Core/TsFile > > Core/Engine > > Core/WAL > > --------------- > > > > What to do you think? > > > > Best, > > Jincheng > > > > > > ------------------------------------------------------ > > 1. --Connectors------------- > > Connectors components include all data integration with external systems, > > The components are listed as follows: > > > > Connectors > > Connectors/Grafana > > Connectors/Spark-TsFile > > Connectors/Spark-IoTDB > > Connectors/Hive-tsfile > > Connectors/Hadoop-tsfile > > > > 2. --Client------------- > > Client components involves RPC between client and server, a command-line > > interface（multi-languages).The components are listed as follows: > > > > Client > > Client/Java > > Client/Python > > > > 3. --Planner------------- > > Planner components involves the query process in IoTDB, including > multiple > > types of queries, such as raw data query, aggregations, group by time, > > group by device, and includes many optimized rules.The components are > > listed as follows: > > > > Planner > > Planner/sql parser > > Planner/sql optimizer > > > > 4. --Core------------- > > Core components is the storage engine of IoTDB. The components are listed > > as follows: > > > > Core > > Core/TsFile > > Core/Engine > > Core/WAL > > > > 5. --Tools------------- > > Tools components involves many utils of IoTDB, Such as data Sync. The > > components are listed as follows: > > Tools > > Tools/Sync > > Tools/Other > > > > 6. --Authority--------- > > Authority > > > > 7. --Distribution------ > > Distribution > > > > 8. --Example----------- > > Example > > > > 9. --WebSite----------- > > WebSite > > > > 10. --Doc--------------- > > Doc > > > > 11. --Other------------ > > Other （I'm not sure if we really need the Other component, but there is > no > > harm in adding one） > > > > Xiangdong Huang <saint...@gmail.com> 于2020年1月17日周五 上午12:50写道： > > > >> Hi, > >> > >> Thanks jincheng and jialin. > >> > >> Sorry for absence of the discussion because of busy work these days.. > >> > >> Some comments: > >> > >> > Core > >> > Core/TsFile > >> > Core/Merge > >> > Core/Storage > >> > Core/WAL > >> > >> Not so clearly.. maybe TsFile, WAL, Storage Engine, Query Engine? (now > >> query engine is in the storage engine..) > >> Merge is a submodule of storage engine and query engine. > >> If we pick out the merge module, we also need to pick out the FLUSH, > >> MODIFICATION, etc.. I think. > >> > >> Best, > >> ----------------------------------- > >> Xiangdong Huang > >> School of Software, Tsinghua University > >> > >> 黄向东 > >> 清华大学 软件学院 > >> > >> > >> Jialin Qiao <qiaojia...@apache.org> 于2020年1月16日周四 下午11:21写道： > >> > >> > Hi Jincheng, > >> > > >> > Thank you! Looks good :) > >> > > >> > jincheng sun <sunjincheng...@gmail.com> 于2020年1月16日周四 下午4:01写道： > >> > > >> > > Thanks for your explanation! @Jialin > >> > > > >> > > I thought about the composition of our components from the external > >> > system, > >> > > the user level, to the IoTDB bottom-level implementation to the > >> auxiliary > >> > > tools, and I adjusted a little based on yours. as follows: > >> > > > >> > > 1. --Connectors------------- > >> > > Connectors components include all data integration with external > >> systems, > >> > > The components are listed as follows: > >> > > > >> > > Connectors > >> > > Connectors/Grafana > >> > > Connectors/Spark-TsFile > >> > > Connectors/Spark-IoTDB > >> > > Connectors/Hive-tsfile > >> > > Connectors/Hadoop-tsfile > >> > > > >> > > 2. --Client------------- > >> > > Client components involves RPC between client and server, a > >> command-line > >> > > interface（multi-languages).The components are listed as follows: > >> > > > >> > > Client > >> > > Client/Java > >> > > Client/Python > >> > > > >> > > 3. --Planner------------- > >> > > Planner components involves the query process in IoTDB, including > >> > multiple > >> > > types of queries, such as raw data query, aggregations, group by > time, > >> > > group by device, and includes many optimized rules.The components > are > >> > > listed as follows: > >> > > > >> > > Planner > >> > > Planner/sql parsar > >> > > Planner/sql optimizer > >> > > > >> > > 4. --Core------------- > >> > > Core components is the storage engine of IoTDB. The components are > >> listed > >> > > as follows: > >> > > > >> > > Core > >> > > Core/TsFile > >> > > Core/Merge > >> > > Core/Storage > >> > > Core/WAL > >> > > > >> > > 5. --Tools------------- > >> > > Tools components involves many utils of IoTDB, Such as data Sync. > The > >> > > components are listed as follows: > >> > > Tools > >> > > Tools/Sync > >> > > Tools/Other > >> > > > >> > > 6. --Authority--------- > >> > > Authority > >> > > > >> > > 7. --Distribution------ > >> > > Distribution > >> > > > >> > > 8. --Example----------- > >> > > Example > >> > > > >> > > 9. --WebSite----------- > >> > > WebSite > >> > > > >> > > 10. --Doc--------------- > >> > > Doc > >> > > > >> > > 11. --Other------------ > >> > > Other （I'm not sure if we really need the Other component, but there > >> is > >> > no > >> > > harm in adding one） > >> > > > >> > > This does not change much from your solution. The core is to add a > >> layer > >> > of > >> > > abstraction. > >> > > > >> > > Does this make sense to you? Anyway we can continue to improve if > >> needed! > >> > > > >> > > Best, > >> > > Jincheng > >> > > > >> > > Jialin Qiao <qiaojia...@apache.org> 于2020年1月15日周三 下午4:28写道： > >> > > > >> > > > Hi, > >> > > > > >> > > > WAL has an independent physical layout that has nothing to do with > >> > > TsFile. > >> > > > It is used in the writing process and recovering in IoTDB. > >> > > > It could be merged into Storage components. > >> > > > > >> > > > I'm not sure whether I leave out some components, so I add an > Other > >> and > >> > > > therefore can not give an example... > >> > > > > >> > > > Thanks, > >> > > > > >> > > > jincheng sun <sunjincheng...@gmail.com> 于2020年1月15日周三 下午4:01写道： > >> > > > > >> > > > > Sorry for the late reply, I just came back from team outing. > >> > > > > > >> > > > > Thanks for adding more detail information. I have two questions > >> then: > >> > > > > > >> > > > > - Is WAL a write strategy for TsFlie, can WAL be incorporated > into > >> > > other > >> > > > > components and what is the reason for being a separate > component? > >> > > > > - Can you give an example of what changes belongs to the Other > >> > > component? > >> > > > > > >> > > > > Best, > >> > > > > Jincheng > >> > > > > > >> > > > > > >> > > > > Jialin Qiao <qiaojia...@apache.org> 于2020年1月9日周四 下午6:39写道： > >> > > > > > >> > > > > > Hi, > >> > > > > > > >> > > > > > Sure, maybe a bit more... > >> > > > > > > >> > > > > > - TsFile : A columnar file format that has flexible schema > and > >> > data > >> > > > > > management for time series. > >> > > > > > - Storage : The storage engine of IoTDB, including writing > to > >> > > > memtable > >> > > > > > (buffer), flushing, system recovery, and memtable management. > >> > > > > > - Query : The query process in IoTDB, including multiple > >> types of > >> > > > > > queries, such as raw data query, aggregations, group by time, > >> group > >> > > by > >> > > > > > device. > >> > > > > > - Merge : Compaction of data files, including multiple > >> compaction > >> > > > > > strategies, such as (1) merging unsequence TsFile with > sequence > >> > > TsFile, > >> > > > > (2) > >> > > > > > merging small sequence TsFiles into a large TsFile. > >> > > > > > - Sync: Synchronize data files from one IoTDB instance to > >> > another. > >> > > > > > - Authority: Administration of IoTDB, including users, roles > >> and > >> > > > > > privileges. > >> > > > > > - WAL : Write-ahead-log. > >> > > > > > - Client: Rpc between client and server, a command-line > >> interface > >> > > and > >> > > > > > client of different languages. > >> > > > > > - Distribution : The shared-nothing distribution of IoTDB, > >> > > including > >> > > > > raft > >> > > > > > protocol, data partition, and system information management. > >> > > > > > - Grafana : The connector for using IoTDB as a data source > in > >> > > > Grafana. > >> > > > > > - Spark-TsFile : A connector that allows users read/write > >> TsFile > >> > > > > through > >> > > > > > SparkSQL (treat TsFile as an external data source for > SparkSQL). > >> > > > > > - Spark-IoTDB : A connector that allows users read/write > IoTDB > >> > > > through > >> > > > > > SparkSQL (treat IoTDB as an external data source for > SparkSQL). > >> > > > > > - Hive-tsfile : A connector that maps TsFile as a table in > >> Hive > >> > for > >> > > > the > >> > > > > > query. > >> > > > > > - Hadoop-tsfile : A connector that allows users read/write > >> TsFile > >> > > > > through > >> > > > > > MapReduce. > >> > > > > > - Tools : Other tools for IoTDB, such as watermark, memory > >> > > > estimation, > >> > > > > > JMX, query history visualization. > >> > > > > > - Example : Examples of different modules, such as JDBC, > >> Session, > >> > > > > TsFile. > >> > > > > > - WebSite : IoTDB official website. > >> > > > > > - Doc: documents of IoTDB, docs folder under root directory. > >> > > > > > - Other: Those do not belong to the above components. > >> > > > > > > >> > > > > > > >> > > > > > jincheng sun <sunjincheng...@gmail.com> 于2020年1月9日周四 > 下午5:28写道： > >> > > > > > > >> > > > > > > Hi, > >> > > > > > > > >> > > > > > > @Xiangdong thank you for helping to clarify my original > >> > intention. > >> > > > you > >> > > > > > are > >> > > > > > > right, I really want to list some components to let users > >> choose > >> > > > which > >> > > > > > > component the given issue belongs to. > >> > > > > > > > >> > > > > > > @Jialin I think approach 4 would be more meaningful for > IoTDB. > >> > > Could > >> > > > > you > >> > > > > > > please add the detailed description for each component, such > >> as, > >> > > > > describe > >> > > > > > > the relationship between `tsfile` and > >> > > > > > > (`tsfile/hive-tsfile/spark-tsfile/Hadoop-tsfile`), and which > >> > > changes > >> > > > > > belong > >> > > > > > > to `Other` component, etc., which can help us better > >> understand > >> > the > >> > > > > > reasons > >> > > > > > > for such classification. On the other hand, we will further > >> > discuss > >> > > > > > whether > >> > > > > > > there is optimization space for this component > classification. > >> > > > > > > > >> > > > > > > What do you think? > >> > > > > > > > >> > > > > > > Best, > >> > > > > > > Jincheng > >> > > > > > > > >> > > > > > > Xiangdong Huang <saint...@gmail.com> 于2020年1月9日周四 下午4:46写道： > >> > > > > > > > >> > > > > > > > > No that is not the Apache Way. No one should be in > >> charge of > >> > > an > >> > > > > area > >> > > > > > > > > >> > > > > > > > I think Jincheng's mean is, we can list some components to > >> let > >> > > > users > >> > > > > > > > choose which component the given issue belongs to. > >> > > > > > > > > >> > > > > > > > As for which components should be listed, we need to have > a > >> > > > > discussion. > >> > > > > > > > > >> > > > > > > > > No one should be in charge of an area > >> > > > > > > > > >> > > > > > > > Yes everyone can assign an issue to himself/herself. > >> > > > > > > > > >> > > > > > > > Best, > >> > > > > > > > ---------------------------------- > >> > > > > > > > Xiangdong Huang > >> > > > > > > > School of Software, Tsinghua University > >> > > > > > > > > >> > > > > > > > 黄向东 > >> > > > > > > > 清华大学 软件学院 > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > jincheng sun <sunjincheng...@gmail.com> 于2020年1月9日周四 > >> 下午4:38写道： > >> > > > > > > > > >> > > > > > > > > > No that is not the Apache Way. No one should be in > >> charge > >> > of > >> > > an > >> > > > > > area. > >> > > > > > > > > > >> > > > > > > > > Oh, great thanks for correcting me Justin, maybe it's my > >> > > language > >> > > > > > > > > expression problems. I mean some contributors and > >> committers > >> > > will > >> > > > > be > >> > > > > > > very > >> > > > > > > > > interested in some components, then spend more time, and > >> be > >> > > more > >> > > > > > > familiar > >> > > > > > > > > with the code. > >> > > > > > > > > > >> > > > > > > > > Best, > >> > > > > > > > > Jincheng > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > Justin Mclean <jus...@classsoftware.com> 于2020年1月9日周四 > >> > > 下午4:27写道： > >> > > > > > > > > > >> > > > > > > > > > Hi, > >> > > > > > > > > > > >> > > > > > > > > > > When our community grows stronger, each\ module may > >> have > >> > > some > >> > > > > > > > specific > >> > > > > > > > > > person in charge. > >> > > > > > > > > > > >> > > > > > > > > > No that is not the Apache Way. No one should be in > >> charge > >> > of > >> > > an > >> > > > > > area. > >> > > > > > > > > > > >> > > > > > > > > > Thanks, > >> > > > > > > > > > Justin > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > -- > >> > > > > > ————————————————— > >> > > > > > Jialin Qiao > >> > > > > > School of Software, Tsinghua University > >> > > > > > > >> > > > > > 乔嘉林 > >> > > > > > 清华大学 软件学院 > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > -- > >> > > > ————————————————— > >> > > > Jialin Qiao > >> > > > School of Software, Tsinghua University > >> > > > > >> > > > 乔嘉林 > >> > > > 清华大学 软件学院 > >> > > > > >> > > > >> > > >> > > >> > -- > >> > ————————————————— > >> > Jialin Qiao > >> > School of Software, Tsinghua University > >> > > >> > 乔嘉林 > >> > 清华大学 软件学院 > >> > > >> > > > -- ----------------------------------- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院
