Hi, 1. InsertTablets could reach more than 3 times faster than insertRecords.
2. Yes, Tablet is actually a small table with some columns and rows. It has a time column and many value columns. In each row, all columns must have a value. An example: time, root.sg.d1.s1, root.sg.d1.s2 1, 1, 2.2 2, 1, 2.2 3, 1, 2.2 Tablets do not allow to have null values. However, the tablet uses an array of primitive types to store data in a columnar format, e.g., long[], int[], double[]. If you store data in a text file, the data type indicator is needed, like this: time, root.sg.d1.s1, root.sg.d1.s2 long, int, double 1, 1, 2.2 2, 1, 2.2 3, 1, 2.2 Then you could generate a tablet from the data and then use the insertTablet. 3. You could check the memory allocated in Raspberry pi. Is it the same as on desktop? This may impact the write throughput. Thanks, ————————————————— Jialin Qiao School of Software, Tsinghua University 乔嘉林 清华大学 软件学院 Dhruv Garg <[email protected]> 于2021年4月8日周四 下午1:36写道： > Hello all, > > In the past month I have been using the JDBC client of IoTDB to write data > from CSV into IoTDB and also query on the data. Looking at the CSV code in > ImportCsv.java > < > https://github.com/apache/iotdb/blob/master/cli/src/main/java/org/apache/iotdb/tool/ImportCsv.java > >, > it seems that csv itself is again parsed into an IoTDB-friendly structure > and then ingested. I would like to avoid the csv-parsing time and directly > provide the data as needed to IoTDB. This should improve the ingestion > performance. > > What I am talking about is similar to InfluxDB where we can also parse from > CSV to InfluxDB's native line protocol and then ingest the data into DB. > However, for better performance, InfluxDB also provides write APIs > < > https://github.com/influxdata/influxdb-client-java/blob/master/client/src/main/java/com/influxdb/client/WriteApi.java > > > to take records in line protocol as input and directly ingest those. > > I have three questions: > > 1. I see that IoTDB is getting newer write APIs like InsertTablets and > it seems that it is designed to be faster than insertRecords. > Approximately > how much performance improvement have you seen with InsertTablets? > 2. Is there a way to create a data file such that it is easy to use > InsertTablets with it? This is to know if I can create an IoTDB-friendly > and IoTDB-specific input file and then directly use InsertTablets to > ingest > the data. > 3. As a preliminary check, I am also trying out IoTDB on Raspberry Pi 4B > devices. However, the ingestion time with CSV on the Raspberry Pi is > taking > 10 times of what it is on the desktop (amd64). This ratio should have > ideally been closer to 4X, based on other applications that I have > benchmarked. Have you all run any Raspberry Pi benchmarks for IoTDB > earlier? > > I would be awaiting your response. Thanks! > > Regards, > dgargcs >
