Hello Jialin, Thanks for your response.
1. Alright I will definitely move from insertRecords() to insertTablet() then. *Can I use the API on v0.11.2 (I am currently using this) or do I need v0.11.3?* 2. The info you provided on insertTablet() is very helpful. I could start with that. I think specifying the data types at the top of the file will also help reduce the data-type inference time for the api and possibly reduce ingestion time. *Is there any limit to the number of rows that can be inserted at once using insertTablet()? Or, is there an optimal number of rows per insertTablet() to get best performance?* 3. Also, I see that there is a new API in v0.12 called insertRecordsOfOneDevice(). *Since insertTablet() also takes rows for one device, what exactly is the difference between insertTablet() and insertRecordsOfOneDevice()? Are they for different use-cases? Which performs better?* 4. So the Raspberry Pi are low-end devices. They run on lower RAM (2GB), have ARM processors and use SD cards for persistent storage. *I suppose IoTDB does not have benchmark numbers for such devices? * A couple of additional questions: 5. I see that the IoTDB team is nearing release of v0.12. *How soon can we expect it to come out? Or can I test it out even now since I eventually plan to start working with it? Is it stable enough?* 6. *What benefits does the new Tsfile structure in v0.12 bring? Does it improve DB data ingestion/query performance?* Thanks in advance! On Thu, 8 Apr 2021 at 17:26, Jialin Qiao <[email protected]> wrote: > Hi, > > 1. InsertTablets could reach more than 3 times faster than insertRecords. > > 2. Yes, Tablet is actually a small table with some columns and rows. > It has a time column and many value columns. In each row, all columns must > have a value. > > An example: > > time, root.sg.d1.s1, root.sg.d1.s2 > 1, 1, 2.2 > 2, 1, 2.2 > 3, 1, 2.2 > > Tablets do not allow to have null values. > > However, the tablet uses an array of primitive types to store data in a > columnar format, e.g., long[], int[], double[]. > If you store data in a text file, the data type indicator is needed, like > this: > > time, root.sg.d1.s1, root.sg.d1.s2 > long, int, double > 1, 1, 2.2 > 2, 1, 2.2 > 3, 1, 2.2 > > Then you could generate a tablet from the data and then use the > insertTablet. > > 3. You could check the memory allocated in Raspberry pi. Is it the same as > on desktop? This may impact the write throughput. > > Thanks, > ————————————————— > Jialin Qiao > School of Software, Tsinghua University > > 乔嘉林 > 清华大学 软件学院 > > > Dhruv Garg <[email protected]> 于2021年4月8日周四 下午1:36写道： > > > Hello all, > > > > In the past month I have been using the JDBC client of IoTDB to write > data > > from CSV into IoTDB and also query on the data. Looking at the CSV code > in > > ImportCsv.java > > < > > > https://github.com/apache/iotdb/blob/master/cli/src/main/java/org/apache/iotdb/tool/ImportCsv.java > > >, > > it seems that csv itself is again parsed into an IoTDB-friendly structure > > and then ingested. I would like to avoid the csv-parsing time and > directly > > provide the data as needed to IoTDB. This should improve the ingestion > > performance. > > > > What I am talking about is similar to InfluxDB where we can also parse > from > > CSV to InfluxDB's native line protocol and then ingest the data into DB. > > However, for better performance, InfluxDB also provides write APIs > > < > > > https://github.com/influxdata/influxdb-client-java/blob/master/client/src/main/java/com/influxdb/client/WriteApi.java > > > > > to take records in line protocol as input and directly ingest those. > > > > I have three questions: > > > > 1. I see that IoTDB is getting newer write APIs like InsertTablets and > > it seems that it is designed to be faster than insertRecords. > > Approximately > > how much performance improvement have you seen with InsertTablets? > > 2. Is there a way to create a data file such that it is easy to use > > InsertTablets with it? This is to know if I can create an > IoTDB-friendly > > and IoTDB-specific input file and then directly use InsertTablets to > > ingest > > the data. > > 3. As a preliminary check, I am also trying out IoTDB on Raspberry Pi > 4B > > devices. However, the ingestion time with CSV on the Raspberry Pi is > > taking > > 10 times of what it is on the desktop (amd64). This ratio should have > > ideally been closer to 4X, based on other applications that I have > > benchmarked. Have you all run any Raspberry Pi benchmarks for IoTDB > > earlier? > > > > I would be awaiting your response. Thanks! > > > > Regards, > > dgargcs > > >
