good idea，i support redesign the csv
---Original--- From: "Xiangdong Huang"<saint...@gmail.com> Date: Wed, Jun 9, 2021 14:19 PM To: "dev"<dev@iotdb.apache.org>; Subject: Re: support csv format Hi, Yes. Do you or your mates have time to do the implementation? I think, 1. we need to modify the document of CSV Tools, and add sample data into your examples folder. 2. rewrite the tool and (only) support RFC 4180 3. redesign the csv schema if needed. Best, ----------------------------------- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Chao Wang <ccgow...@163.com> 于2021年6月9日周三 上午9:17写道： > > Good idea! > > > It's friendly to user for supporting `align by device` schema. > > > And I think the csv format is also important, we could pre-define a standard csv format to make user know not all csv format we could support. > > > If we don't pass this kind of information to them, the user will think that all types are supported, and it's not clear how to modify when the import error occurs, so as to ensure that the data can be imported normally. > > > | > Chao Wang > ccgow...@163.com > | > 签名由网易邮箱大师定制 > On 6/9/2021 08:58，Xiangdong Huang<saint...@gmail.com> wrote： > Hi, > > At present, I see that the import CSV tool of iotdb does not support the file format of CSV very clearly. And users will be very confused about using the tool. > IMO, a standard CSV format is needed. Apache common CSV is fine, RFC > 4180 is fine. > BUT, what is the real confusing thing? whether the CSV obeys RFC > 4180? or what does the schema of the CSV is? > > I think the answer is the latter. > > For example, as far as I know, current CSV schema requires like this: > ``` > Time, root.d1.s1, root.d1.s2, root.d2.s1, root.d2.s5 ... > ``` > (also called, `align by time` table format) > > Is this really good for users? And is this really good for developers > (IoTDB developers, I mean)? > > - We can say YES, because it supports different devices having > different measurements/sensors. > > - We can say NO, because in a Relational database user's view, the > following table schema is more comfortable: > > ``` > Time, deviceId, s1, s2, s3 ... > ``` > (also called, `align by device` table format) > > Meanwhile, the above schema will make CSV processing much easier for us. > E.g., we can use insertRecordsOfOneDevice, we can use Tablets. > If we require the data in the CSV must be ordered in <deviceID, time>, > then we can do more. > > (More about CSV Tool, which has no relation with the above two schemas: > If we improve our TsFile Load tool, we can even convert the CSV to a > TsFile directly and then load it to IoTDB.) > > > So, I think the more important thing is, which schema we need to support in CSV? > Maybe this question can be extended to another big question: which > kind of relational table structure we want to support (or want IoTDB > can convert to)? > > Considering many users want to integrate IoTDB with existing RDB tools/JDBC. > I think maybe we need to make a decision. That is, if we want to > convert IoTDB to a relational table, just use `align by device` table > format. > > 1. For csv, we just export data from IoTDB to `align by device` table > format. And by default, we accept CSV that obeys this format. (Support > format 1 can also be considered). > > 2. For JDBC (which I want to consider together), Add a parameter in to > the JDBC URL: jdbc:iotdb://<IP>:<port>?alignByDevice=true > If `alignByDevice=true`, then we support all JDBC interfaces, e.g., > getTableName, etc.. and try to support DBeaver and so on. > If there is no such a parameter, we still throw `not support` for many > JDBC interfaces. > > At end, THIS IS JUST AN IDEA. Requires more discussion. > > Best, > ----------------------------------- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > Chao Wang <ccgow...@163.com> 于2021年6月8日周二 下午4:12写道： > > Hi, > At present, I see that the import CSV tool of iotdb does not support the file format of CSV very clearly. And users will be very confused about using the tool. > I notice that Apache common CSV is a well-known project, which supports a variety of popular CSV formats in the industry. For example, Excel, mysql, rfc4180, Oracle, PostgreSQL_ CSV。 > > > Can we pre define what type of CSV format we want to support, such as rfc4180? > > > | > Chao Wang > ccgow...@163.com > | > 签名由网易邮箱大师定制
