I would like to bring up a VOTE mail to make the final decision If there no objections.
What do you think? Best, Jincheng ------------- Twitter: https://twitter.com/sunjincheng121 ------------- jincheng sun <sunjincheng...@gmail.com> 于2020年1月17日周五 上午10:39写道： > Great thanks for adding more relationship info about `Merge` and > `Storage`. @Xiangdong ! > > As : > > (now query engine is in the storage engine..) Merge is a submodule of > storage engine and query engine. > > I think `query engine` and `storage engine` is make sense to me. Here only > a suggestion that how about use the `Engine` for both `Storage Engine` and > `Query Engine` as they are very close each other and they also have the > common share `Merge`. i.e.,: > > -------- > Core > Core/TsFile > Core/Engine > Core/WAL > --------------- > > What to do you think? > > Best, > Jincheng > > > ------------------------------------------------------ > 1. --Connectors------------- > Connectors components include all data integration with external systems, > The components are listed as follows: > > Connectors > Connectors/Grafana > Connectors/Spark-TsFile > Connectors/Spark-IoTDB > Connectors/Hive-tsfile > Connectors/Hadoop-tsfile > > 2. --Client------------- > Client components involves RPC between client and server, a command-line > interface（multi-languages).The components are listed as follows: > > Client > Client/Java > Client/Python > > 3. --Planner------------- > Planner components involves the query process in IoTDB, including multiple > types of queries, such as raw data query, aggregations, group by time, > group by device, and includes many optimized rules.The components are > listed as follows: > > Planner > Planner/sql parser > Planner/sql optimizer > > 4. --Core------------- > Core components is the storage engine of IoTDB. The components are listed > as follows: > > Core > Core/TsFile > Core/Engine > Core/WAL > > 5. --Tools------------- > Tools components involves many utils of IoTDB, Such as data Sync. The > components are listed as follows: > Tools > Tools/Sync > Tools/Other > > 6. --Authority--------- > Authority > > 7. --Distribution------ > Distribution > > 8. --Example----------- > Example > > 9. --WebSite----------- > WebSite > > 10. --Doc--------------- > Doc > > 11. --Other------------ > Other （I'm not sure if we really need the Other component, but there is no > harm in adding one） > > Xiangdong Huang <saint...@gmail.com> 于2020年1月17日周五 上午12:50写道： > >> Hi, >> >> Thanks jincheng and jialin. >> >> Sorry for absence of the discussion because of busy work these days.. >> >> Some comments: >> >> > Core >> > Core/TsFile >> > Core/Merge >> > Core/Storage >> > Core/WAL >> >> Not so clearly.. maybe TsFile, WAL, Storage Engine, Query Engine? (now >> query engine is in the storage engine..) >> Merge is a submodule of storage engine and query engine. >> If we pick out the merge module, we also need to pick out the FLUSH, >> MODIFICATION, etc.. I think. >> >> Best, >> ----------------------------------- >> Xiangdong Huang >> School of Software, Tsinghua University >> >> 黄向东 >> 清华大学 软件学院 >> >> >> Jialin Qiao <qiaojia...@apache.org> 于2020年1月16日周四 下午11:21写道： >> >> > Hi Jincheng, >> > >> > Thank you! Looks good :) >> > >> > jincheng sun <sunjincheng...@gmail.com> 于2020年1月16日周四 下午4:01写道： >> > >> > > Thanks for your explanation! @Jialin >> > > >> > > I thought about the composition of our components from the external >> > system, >> > > the user level, to the IoTDB bottom-level implementation to the >> auxiliary >> > > tools, and I adjusted a little based on yours. as follows: >> > > >> > > 1. --Connectors------------- >> > > Connectors components include all data integration with external >> systems, >> > > The components are listed as follows: >> > > >> > > Connectors >> > > Connectors/Grafana >> > > Connectors/Spark-TsFile >> > > Connectors/Spark-IoTDB >> > > Connectors/Hive-tsfile >> > > Connectors/Hadoop-tsfile >> > > >> > > 2. --Client------------- >> > > Client components involves RPC between client and server, a >> command-line >> > > interface（multi-languages).The components are listed as follows: >> > > >> > > Client >> > > Client/Java >> > > Client/Python >> > > >> > > 3. --Planner------------- >> > > Planner components involves the query process in IoTDB, including >> > multiple >> > > types of queries, such as raw data query, aggregations, group by time, >> > > group by device, and includes many optimized rules.The components are >> > > listed as follows: >> > > >> > > Planner >> > > Planner/sql parsar >> > > Planner/sql optimizer >> > > >> > > 4. --Core------------- >> > > Core components is the storage engine of IoTDB. The components are >> listed >> > > as follows: >> > > >> > > Core >> > > Core/TsFile >> > > Core/Merge >> > > Core/Storage >> > > Core/WAL >> > > >> > > 5. --Tools------------- >> > > Tools components involves many utils of IoTDB, Such as data Sync. The >> > > components are listed as follows: >> > > Tools >> > > Tools/Sync >> > > Tools/Other >> > > >> > > 6. --Authority--------- >> > > Authority >> > > >> > > 7. --Distribution------ >> > > Distribution >> > > >> > > 8. --Example----------- >> > > Example >> > > >> > > 9. --WebSite----------- >> > > WebSite >> > > >> > > 10. --Doc--------------- >> > > Doc >> > > >> > > 11. --Other------------ >> > > Other （I'm not sure if we really need the Other component, but there >> is >> > no >> > > harm in adding one） >> > > >> > > This does not change much from your solution. The core is to add a >> layer >> > of >> > > abstraction. >> > > >> > > Does this make sense to you? Anyway we can continue to improve if >> needed! >> > > >> > > Best, >> > > Jincheng >> > > >> > > Jialin Qiao <qiaojia...@apache.org> 于2020年1月15日周三 下午4:28写道： >> > > >> > > > Hi, >> > > > >> > > > WAL has an independent physical layout that has nothing to do with >> > > TsFile. >> > > > It is used in the writing process and recovering in IoTDB. >> > > > It could be merged into Storage components. >> > > > >> > > > I'm not sure whether I leave out some components, so I add an Other >> and >> > > > therefore can not give an example... >> > > > >> > > > Thanks, >> > > > >> > > > jincheng sun <sunjincheng...@gmail.com> 于2020年1月15日周三 下午4:01写道： >> > > > >> > > > > Sorry for the late reply, I just came back from team outing. >> > > > > >> > > > > Thanks for adding more detail information. I have two questions >> then: >> > > > > >> > > > > - Is WAL a write strategy for TsFlie, can WAL be incorporated into >> > > other >> > > > > components and what is the reason for being a separate component? >> > > > > - Can you give an example of what changes belongs to the Other >> > > component? >> > > > > >> > > > > Best, >> > > > > Jincheng >> > > > > >> > > > > >> > > > > Jialin Qiao <qiaojia...@apache.org> 于2020年1月9日周四 下午6:39写道： >> > > > > >> > > > > > Hi, >> > > > > > >> > > > > > Sure, maybe a bit more... >> > > > > > >> > > > > > - TsFile : A columnar file format that has flexible schema and >> > data >> > > > > > management for time series. >> > > > > > - Storage : The storage engine of IoTDB, including writing to >> > > > memtable >> > > > > > (buffer), flushing, system recovery, and memtable management. >> > > > > > - Query : The query process in IoTDB, including multiple >> types of >> > > > > > queries, such as raw data query, aggregations, group by time, >> group >> > > by >> > > > > > device. >> > > > > > - Merge : Compaction of data files, including multiple >> compaction >> > > > > > strategies, such as (1) merging unsequence TsFile with sequence >> > > TsFile, >> > > > > (2) >> > > > > > merging small sequence TsFiles into a large TsFile. >> > > > > > - Sync: Synchronize data files from one IoTDB instance to >> > another. >> > > > > > - Authority: Administration of IoTDB, including users, roles >> and >> > > > > > privileges. >> > > > > > - WAL : Write-ahead-log. >> > > > > > - Client: Rpc between client and server, a command-line >> interface >> > > and >> > > > > > client of different languages. >> > > > > > - Distribution : The shared-nothing distribution of IoTDB, >> > > including >> > > > > raft >> > > > > > protocol, data partition, and system information management. >> > > > > > - Grafana : The connector for using IoTDB as a data source in >> > > > Grafana. >> > > > > > - Spark-TsFile : A connector that allows users read/write >> TsFile >> > > > > through >> > > > > > SparkSQL (treat TsFile as an external data source for SparkSQL). >> > > > > > - Spark-IoTDB : A connector that allows users read/write IoTDB >> > > > through >> > > > > > SparkSQL (treat IoTDB as an external data source for SparkSQL). >> > > > > > - Hive-tsfile : A connector that maps TsFile as a table in >> Hive >> > for >> > > > the >> > > > > > query. >> > > > > > - Hadoop-tsfile : A connector that allows users read/write >> TsFile >> > > > > through >> > > > > > MapReduce. >> > > > > > - Tools : Other tools for IoTDB, such as watermark, memory >> > > > estimation, >> > > > > > JMX, query history visualization. >> > > > > > - Example : Examples of different modules, such as JDBC, >> Session, >> > > > > TsFile. >> > > > > > - WebSite : IoTDB official website. >> > > > > > - Doc: documents of IoTDB, docs folder under root directory. >> > > > > > - Other: Those do not belong to the above components. >> > > > > > >> > > > > > >> > > > > > jincheng sun <sunjincheng...@gmail.com> 于2020年1月9日周四 下午5:28写道： >> > > > > > >> > > > > > > Hi, >> > > > > > > >> > > > > > > @Xiangdong thank you for helping to clarify my original >> > intention. >> > > > you >> > > > > > are >> > > > > > > right, I really want to list some components to let users >> choose >> > > > which >> > > > > > > component the given issue belongs to. >> > > > > > > >> > > > > > > @Jialin I think approach 4 would be more meaningful for IoTDB. >> > > Could >> > > > > you >> > > > > > > please add the detailed description for each component, such >> as, >> > > > > describe >> > > > > > > the relationship between `tsfile` and >> > > > > > > (`tsfile/hive-tsfile/spark-tsfile/Hadoop-tsfile`), and which >> > > changes >> > > > > > belong >> > > > > > > to `Other` component, etc., which can help us better >> understand >> > the >> > > > > > reasons >> > > > > > > for such classification. On the other hand, we will further >> > discuss >> > > > > > whether >> > > > > > > there is optimization space for this component classification. >> > > > > > > >> > > > > > > What do you think? >> > > > > > > >> > > > > > > Best, >> > > > > > > Jincheng >> > > > > > > >> > > > > > > Xiangdong Huang <saint...@gmail.com> 于2020年1月9日周四 下午4:46写道： >> > > > > > > >> > > > > > > > > No that is not the Apache Way. No one should be in >> charge of >> > > an >> > > > > area >> > > > > > > > >> > > > > > > > I think Jincheng's mean is, we can list some components to >> let >> > > > users >> > > > > > > > choose which component the given issue belongs to. >> > > > > > > > >> > > > > > > > As for which components should be listed, we need to have a >> > > > > discussion. >> > > > > > > > >> > > > > > > > > No one should be in charge of an area >> > > > > > > > >> > > > > > > > Yes everyone can assign an issue to himself/herself. >> > > > > > > > >> > > > > > > > Best, >> > > > > > > > ---------------------------------- >> > > > > > > > Xiangdong Huang >> > > > > > > > School of Software, Tsinghua University >> > > > > > > > >> > > > > > > > 黄向东 >> > > > > > > > 清华大学 软件学院 >> > > > > > > > >> > > > > > > > >> > > > > > > > jincheng sun <sunjincheng...@gmail.com> 于2020年1月9日周四 >> 下午4:38写道： >> > > > > > > > >> > > > > > > > > > No that is not the Apache Way. No one should be in >> charge >> > of >> > > an >> > > > > > area. >> > > > > > > > > >> > > > > > > > > Oh, great thanks for correcting me Justin, maybe it's my >> > > language >> > > > > > > > > expression problems. I mean some contributors and >> committers >> > > will >> > > > > be >> > > > > > > very >> > > > > > > > > interested in some components, then spend more time, and >> be >> > > more >> > > > > > > familiar >> > > > > > > > > with the code. >> > > > > > > > > >> > > > > > > > > Best, >> > > > > > > > > Jincheng >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > Justin Mclean <jus...@classsoftware.com> 于2020年1月9日周四 >> > > 下午4:27写道： >> > > > > > > > > >> > > > > > > > > > Hi, >> > > > > > > > > > >> > > > > > > > > > > When our community grows stronger, each\ module may >> have >> > > some >> > > > > > > > specific >> > > > > > > > > > person in charge. >> > > > > > > > > > >> > > > > > > > > > No that is not the Apache Way. No one should be in >> charge >> > of >> > > an >> > > > > > area. >> > > > > > > > > > >> > > > > > > > > > Thanks, >> > > > > > > > > > Justin >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > ————————————————— >> > > > > > Jialin Qiao >> > > > > > School of Software, Tsinghua University >> > > > > > >> > > > > > 乔嘉林 >> > > > > > 清华大学 软件学院 >> > > > > > >> > > > > >> > > > >> > > > >> > > > -- >> > > > ————————————————— >> > > > Jialin Qiao >> > > > School of Software, Tsinghua University >> > > > >> > > > 乔嘉林 >> > > > 清华大学 软件学院 >> > > > >> > > >> > >> > >> > -- >> > ————————————————— >> > Jialin Qiao >> > School of Software, Tsinghua University >> > >> > 乔嘉林 >> > 清华大学 软件学院 >> > >> >
