This is an automated email from the ASF dual-hosted git repository.
tanxinyu pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iotdb-docs.git
The following commit(s) were added to refs/heads/main by this push:
new dbca599 Add Data-Import-Export-Tool (#288)
dbca599 is described below
commit dbca599cc47391b09a698f9b3d1fa7b04efd9e30
Author: Summer <[email protected]>
AuthorDate: Fri Aug 2 17:34:39 2024 +0800
Add Data-Import-Export-Tool (#288)
* Add Data-Import-Export-Tool
* update md name from Import-Export-Tool to TsFile-Import-Export-Tool
* update md title Tool to Script
* batch csv use new md link
---------
Co-authored-by: 2b3c511 <[email protected]>
---
src/.vuepress/sidebar/V1.3.x/en.ts | 3 +-
src/.vuepress/sidebar/V1.3.x/zh.ts | 3 +-
src/.vuepress/sidebar_timecho/V1.3.x/en.ts | 3 +-
src/.vuepress/sidebar_timecho/V1.3.x/zh.ts | 3 +-
src/UserGuide/Master/SQL-Manual/SQL-Manual.md | 2 +-
.../Master/Tools-System/Data-Import-Export-Tool.md | 278 +++++++++++++++++++++
...Export-Tool.md => TsFile-Import-Export-Tool.md} | 4 +-
.../Master/User-Manual/Write-Delete-Data.md | 4 +-
src/UserGuide/latest/SQL-Manual/SQL-Manual.md | 2 +-
.../latest/Tools-System/Data-Import-Export-Tool.md | 278 +++++++++++++++++++++
...Export-Tool.md => TsFile-Import-Export-Tool.md} | 4 +-
.../latest/User-Manual/Write-Delete-Data.md | 4 +-
src/zh/Download/README.md | 2 +-
.../Master/Tools-System/Data-Import-Export-Tool.md | 278 +++++++++++++++++++++
...Export-Tool.md => TsFile-Import-Export-Tool.md} | 6 +-
.../Master/User-Manual/Write-Delete-Data.md | 4 +-
.../latest/Tools-System/Data-Import-Export-Tool.md | 278 +++++++++++++++++++++
...Export-Tool.md => TsFile-Import-Export-Tool.md} | 6 +-
.../latest/User-Manual/Write-Delete-Data.md | 4 +-
19 files changed, 1141 insertions(+), 25 deletions(-)
diff --git a/src/.vuepress/sidebar/V1.3.x/en.ts
b/src/.vuepress/sidebar/V1.3.x/en.ts
index 14d53e2..4d591c5 100644
--- a/src/.vuepress/sidebar/V1.3.x/en.ts
+++ b/src/.vuepress/sidebar/V1.3.x/en.ts
@@ -104,7 +104,8 @@ export const enSidebar = {
{ text: 'Monitor Tool', link: 'Monitor-Tool' },
{ text: 'Benchmark', link: 'Benchmark' },
{ text: 'Maintenance Tool', link: 'Maintenance-Tool' },
- { text: 'Import Export Tool', link: 'Import-Export-Tool' },
+ { text: 'Data Import Export Script', link: 'Data-Import-Export-Tool' },
+ { text: 'TsFile Import Export Script', link:
'TsFile-Import-Export-Tool' },
],
},
{
diff --git a/src/.vuepress/sidebar/V1.3.x/zh.ts
b/src/.vuepress/sidebar/V1.3.x/zh.ts
index 3ac77ae..5645c61 100644
--- a/src/.vuepress/sidebar/V1.3.x/zh.ts
+++ b/src/.vuepress/sidebar/V1.3.x/zh.ts
@@ -104,7 +104,8 @@ export const zhSidebar = {
{ text: '监控工具', link: 'Monitor-Tool' },
{ text: '测试工具', link: 'Benchmark' },
{ text: '运维工具', link: 'Maintenance-Tool' },
- { text: '导入导出工具', link: 'Import-Export-Tool' },
+ { text: '数据导入导出脚本', link: 'Data-Import-Export-Tool' },
+ { text: 'TsFile导入导出脚本', link: 'TsFile-Import-Export-Tool' },
],
},
{
diff --git a/src/.vuepress/sidebar_timecho/V1.3.x/en.ts
b/src/.vuepress/sidebar_timecho/V1.3.x/en.ts
index b34c069..cc9a890 100644
--- a/src/.vuepress/sidebar_timecho/V1.3.x/en.ts
+++ b/src/.vuepress/sidebar_timecho/V1.3.x/en.ts
@@ -113,7 +113,8 @@ export const enSidebar = {
{ text: 'Monitor Tool', link: 'Monitor-Tool_timecho' },
{ text: 'Benchmark', link: 'Benchmark' },
{ text: 'Maintenance Tool', link: 'Maintenance-Tool_timecho' },
- { text: 'Import Export Tool', link: 'Import-Export-Tool' },
+ { text: 'Data Import Export Script', link: 'Data-Import-Export-Tool' },
+ { text: 'TsFile Import Export Script', link:
'TsFile-Import-Export-Tool' },
],
},
{
diff --git a/src/.vuepress/sidebar_timecho/V1.3.x/zh.ts
b/src/.vuepress/sidebar_timecho/V1.3.x/zh.ts
index 1ea41fd..2b437ea 100644
--- a/src/.vuepress/sidebar_timecho/V1.3.x/zh.ts
+++ b/src/.vuepress/sidebar_timecho/V1.3.x/zh.ts
@@ -113,7 +113,8 @@ export const zhSidebar = {
{ text: '监控工具', link: 'Monitor-Tool_timecho' },
{ text: '测试工具', link: 'Benchmark' },
{ text: '运维工具', link: 'Maintenance-Tool_timecho' },
- { text: '导入导出工具', link: 'Import-Export-Tool' },
+ { text: '数据导入导出脚本', link: 'Data-Import-Export-Tool' },
+ { text: 'TsFile导入导出脚本', link: 'TsFile-Import-Export-Tool' },
],
},
{
diff --git a/src/UserGuide/Master/SQL-Manual/SQL-Manual.md
b/src/UserGuide/Master/SQL-Manual/SQL-Manual.md
index 44e105c..d7d576d 100644
--- a/src/UserGuide/Master/SQL-Manual/SQL-Manual.md
+++ b/src/UserGuide/Master/SQL-Manual/SQL-Manual.md
@@ -443,7 +443,7 @@ IoTDB > select * from root.sg1.d1
### Load External TsFile Tool
-For more details, see document
[Import-Export-Tool](../Tools-System/Import-Export-Tool.md).
+For more details, see document
[Import-Export-Tool](../Tools-System/TsFile-Import-Export-Tool.md).
#### Load with SQL
diff --git a/src/UserGuide/Master/Tools-System/Data-Import-Export-Tool.md
b/src/UserGuide/Master/Tools-System/Data-Import-Export-Tool.md
new file mode 100644
index 0000000..c84fb06
--- /dev/null
+++ b/src/UserGuide/Master/Tools-System/Data-Import-Export-Tool.md
@@ -0,0 +1,278 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# Data Import Export Script
+
+IoTDB provides data import and export scripts (tools/export-data,
tools/import-data, supported in versions 1.3.2 and above; for historical
versions, tools/export-csv, tools/import-csv scripts can be used, see the
reference link for usage [Import Export Tool](./TsFile-Import-Export-Tool.md)
), which are used to facilitate the interaction between IoTDB internal data and
external files, suitable for batch operations of single files or directories.
+
+
+## Supported Data Formats
+
+- **CSV** : Plain text format for storing formatted data, which must be
constructed according to the specified CSV format below.
+- **SQL** : Files containing custom SQL statements.
+
+# export-data Script (Data Export)
+
+## Command
+
+```Bash
+# Unix/OS X
+>tools/export-data.sh -h <ip> -p <port> -u <username> -pw <password> -t
<directory> [-tf <time-format> -datatype <true/false> -q <query command> -s
<source sql file> -tfn <target file name> -lpf <int> -type <export type>
-aligned <true/false>]
+
+# Windows
+>tools\export-data.bat -h <ip> -p <port> -u <username> -pw <password> -t
<directory> [-tf <time-format> -datatype <true/false> -q <query command> -s
<source sql file> -tfn <target file name> -lpf <int> -type <export type>
-aligned <true/false>]
+```
+
+Parameter Introduction:
+
+| Parameter | Definition
| Required | Default
|
+|:-------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------|:-------------------------|
+| -h | Database IP address
| No | 127.0.0.1
|
+| -p | Database port
| No | 6667
|
+| -u | Database connection username
| No | root
|
+| -pw | Database connection password
| No | root
|
+| -t | Output path for the exported CSV or SQL file
| Yes |
|
+| -datatype | Whether to print the corresponding data type behind the time
series in the CSV file header, options are true or false
| No | true
|
+| -q | Directly specify the query statement to be executed in the
command (currently only supports some statements, see the table below for
details). <br/>Note: -q and -s parameters must be filled in one, and -q takes
effect if both are filled. For detailed examples of supported SQL statements,
please refer to the "SQL Statement Support Details" below.
| No |
|
+| -s | Specify an SQL file, which may contain one or more SQL
statements. If there are multiple SQL statements, they should be separated by
newlines (returns). Each SQL statement corresponds to one or more output CSV or
SQL files. <br/>Note: -q and -s parameters must be filled in one, and -q takes
effect if both are filled. For detailed examples of supported SQL statements,
please refer to the "SQL Statement Support Details" below. | No |
|
+| -type | Specify the type of exported file, options are csv or sql
| No | csv
|
+| -tf | Specify the time format. The time format must comply with the
[ISO 8601](https://calendars.wikia.org/wiki/ISO_8601) standard or timestamp.
<br/>Note: Only effective when -type is csv
| No | yyyy-MM-dd
HH:mm:ss.SSSz |
+| -lpf | Specify the maximum number of lines for the exported dump
file
| No | 10000
|
+| -timeout | Specify the timeout time for session queries in milliseconds
| No | -1
|
+
+SQL Statement Support Rules:
+
+1. Only query statements are supported; non-query statements (such as metadata
management, system management, etc.) are not supported. For unsupported SQL,
the program will automatically skip and output an error message.
+2. In the current version of query statements, only the export of raw data is
supported. If there are group by, aggregate functions, UDFs, or operational
operators, they are not supported for export as SQL. When exporting raw data,
please note that if exporting data from multiple devices, please use the align
by device statement. Detailed examples are as follows:
+
+| | Supported for
Export | Example |
+|--------------------------------------------------------|----------------------|-----------------------------------------------|
+| Raw data single device query | Supported
| select * from root.s_0.d_0 |
+| Raw data multi-device query (align by device) | Supported
| select * from root.** align by device |
+| Raw data multi-device query (without align by device) | Unsupported
| select * from root.**<br/>select * from root.s_0.* |
+
+## Running Examples
+
+- Export all data within a certain SQL execution range to a CSV file.
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -t ./data/ -q 'select * from root.stock.**'
+ # Windows
+ >tools/export-data.bat -t ./data/ -q 'select * from root.stock.**'
+ ```
+
+- Export Results
+ ```Bash
+
Time,root.stock.Legacy.0700HK.L1_BidPrice,root.stock.Legacy.0700HK.Type,root.stock.Legacy.0700HK.L1_BidSize,root.stock.Legacy.0700HK.Domain,root.stock.Legacy.0700HK.L1_BuyNo,root.stock.Legacy.0700HK.L1_AskPrice
+
2024-07-29T18:37:18.700+08:00,0.9666617,3.0,0.021367407654674264,-6.0,false,0.8926191
+
2024-07-29T18:37:19.701+08:00,0.3057328,3.0,0.9965377284981661,-5.0,false,0.15167356
+ ```
+- All data within the scope of all SQL statements in the SQL file is exported
to CSV files.
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -t ./data/ -s export.sql
+ # Windows
+ >tools/export-data.bat -t ./data/ -s export.sql
+ ```
+
+- Contents of export.sql File (Pointed to by -s Parameter)
+ ```SQL
+ select * from root.stock.** limit 100
+ select * from root.db.** limit 100
+ ```
+
+- Export Result File 1
+ ```Bash
+
Time,root.stock.Legacy.0700HK.L1_BidPrice,root.stock.Legacy.0700HK.Type,root.stock.Legacy.0700HK.L1_BidSize,root.stock.Legacy.0700HK.Domain,root.stock.Legacy.0700HK.L1_BuyNo,root.stock.Legacy.0700HK.L1_AskPrice
+
2024-07-29T18:37:18.700+08:00,0.9666617,3.0,0.021367407654674264,-6.0,false,0.8926191
+
2024-07-29T18:37:19.701+08:00,0.3057328,3.0,0.9965377284981661,-5.0,false,0.15167356
+ ```
+
+- Export Result File 2
+ ```Bash
+ Time,root.db.Random.RandomBoolean
+ 2024-07-22T17:16:05.820+08:00,true
+ 2024-07-22T17:16:02.597+08:00,false
+ ```
+- Export Data in SQL File to SQL Statements with Aligned Format
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -h 127.0.0.1 -p 6667 -u root -p root -t ./data/ -s
export.sql -type sql -aligned true
+ # Windows
+ >tools/export-data.bat -h 127.0.0.1 -p 6667 -u root -p root -t ./data/ -s
export.sql -type sql -aligned true
+ ```
+
+- Export Results
+ ```Bash
+ INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
ALIGNED VALUES
(1722249629831,0.62308747,2.0,0.012206747854849653,-6.0,false,0.14164352);
+ INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
ALIGNED VALUES
(1722249630834,0.7520042,3.0,0.22760657101910464,-5.0,true,0.089064896);
+ INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
ALIGNED VALUES
(1722249631835,0.3981064,3.0,0.6254559288663467,-6.0,false,0.9767922);
+ ```
+- Export Data in a Certain SQL Execution Range to a CSV File with Specified
Time Format and Data Types
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -t ./data/ -tf 'yyyy-MM-dd HH:mm:ss' -datatype true
-q "select * from root.stock.**" -type csv
+ # Windows
+ >tools/export-data.bat -t ./data/ -tf 'yyyy-MM-dd HH:mm:ss' -datatype true
-q "select * from root.stock.**" -type csv
+ ```
+
+- Export Results
+ ```Bash
+
Time,root.stock.Legacy.0700HK.L1_BidPrice(DOUBLE),root.stock.Legacy.0700HK.Type(DOUBLE),root.stock.Legacy.0700HK.L1_BidSize(DOUBLE),root.stock.Legacy.0700HK.Domain(DOUBLE),root.stock.Legacy.0700HK.L1_BuyNo(BOOLEAN),root.stock.Legacy.0700HK.L1_AskPrice(DOUBLE)
+ 2024-07-30 10:33:55,0.44574088,3.0,0.21476832811611501,-4.0,true,0.5951748
+ 2024-07-30 10:33:56,0.6880933,3.0,0.6289119476165305,-5.0,false,0.114634395
+ ```
+
+# import-data Script (Data Import)
+
+## Import File Examples
+
+### CSV File Example
+
+Note that before importing CSV data, special characters need to be handled as
follows:
+
+1. If the text type field contains special characters such as `,`, it should
be escaped with `\`.
+2. You can import times in formats like `yyyy-MM-dd'T'HH:mm:ss`, `yyyy-MM-dd
HH:mm:ss`, or `yyyy-MM-dd'T'HH:mm:ss.SSSZ`.
+3. The time column `Time` should always be in the first column.
+
+Example 1: Time Aligned, No Data Types in Header
+
+```SQL
+Time,root.test.t1.str,root.test.t2.str,root.test.t2.var
+1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
+1970-01-01T08:00:00.002+08:00,"123",,
+```
+
+Example 2: Time Aligned, Data Types in Header(Text type data supports double
quotation marks and non double quotation marks)
+
+```SQL
+Time,root.test.t1.str(TEXT),root.test.t2.str(TEXT),root.test.t2.var(INT32)
+1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
+1970-01-01T08:00:00.002+08:00,123,hello world,123
+1970-01-01T08:00:00.003+08:00,"123",,
+1970-01-01T08:00:00.004+08:00,123,,12
+```
+Example 3: Device Aligned, No Data Types in Header
+
+```SQL
+Time,Device,str,var
+1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
+1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
+1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
+```
+
+Example 4: Device Aligned, Data Types in Header (Text type data supports
double quotation marks and non double quotation marks)
+
+```SQL
+Time,Device,str(TEXT),var(INT32)
+1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
+1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
+1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
+1970-01-01T08:00:00.002+08:00,root.test.t1,hello world,123
+```
+
+### SQL File Example
+
+> For unsupported SQL, illegal SQL, or failed SQL executions, they will be
placed in the failed directory under the failed file (default to
filename.failed).
+
+```SQL
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728578812,0.21911979,4.0,0.7129878488375604,-5.0,false,0.65362453);
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728579812,0.35814416,3.0,0.04674720094979623,-5.0,false,0.9365247);
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728580813,0.20012152,3.0,0.9910098187911393,-4.0,true,0.70040536);
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728581814,0.034122765,4.0,0.9313345284181858,-4.0,true,0.9945297);
+```
+
+## Command
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -h <ip> -p <port> -u <username> -pw <password> -s
<xxx.csv/sql> [-fd <./failedDirectory> -aligned <true/false> -batch <int> -tp
<ms/ns/us> -typeInfer <boolean=text,float=double...> -lpf <int>]
+
+# Windows
+>tools\import-data.bat -h <ip> -p <port> -u <username> -pw <password> -s
<xxx.csv/sql> [-fd <./failedDirectory> -aligned <true/false> -batch <int> -tp
<ms/ns/us> -typeInfer <boolean=text,float=double...> -lpf <int>]
+```
+
+> Although IoTDB has the ability to infer types, it is still recommended to
create metadata before importing data to avoid unnecessary type conversion
errors. For example:
+
+```SQL
+CREATE DATABASE root.fit.d1;
+CREATE DATABASE root.fit.d2;
+CREATE DATABASE root.fit.p;
+CREATE TIMESERIES root.fit.d1.s1 WITH DATATYPE=INT32,ENCODING=RLE;
+CREATE TIMESERIES root.fit.d1.s2 WITH DATATYPE=TEXT,ENCODING=PLAIN;
+CREATE TIMESERIES root.fit.d2.s1 WITH DATATYPE=INT32,ENCODING=RLE;
+CREATE TIMESERIES root.fit.d2.s3 WITH DATATYPE=INT32,ENCODING=RLE;
+CREATE TIMESERIES root.fit.p.s1 WITH DATATYPE=INT32,ENCODING=RLE;
+```
+
+Parameter Introduction:
+
+| Parameter
| Definition
[...]
+|:-----------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| -h
| Database IP address
[...]
+| -p
| Database port
[...]
+| -u
| Database connection username
[...]
+| -pw
| Database connection password
[...]
+| -s
| Specify the data you want to import, here you can specify a file or folder.
If a folder is specified, all files with the suffix CSV or SQL in the folder
will be imported in bulk.
[...]
+| -fd
| Specify the directory to store the failed SQL files. If this parameter is not
specified, the failed files will be saved to the directory of the source data.
<br/>Note: For unsupported SQL, illegal SQL, and failed SQL, they will be
placed in the failed file in the failed directory (default file name is.
failed)
[...]
+| -aligned
| Specify whether to use the 'aligned' interface, with options of true or
false. <br/>Note: This parameter only takes effect when the imported file is a
CSV file
[...]
+| -batch
| Used to specify the number of points to be inserted for each batch of data
(minimum value is 1, maximum value is Integer.MAX_VALUE). If the program
reports' org.apache.hrift.transport ' If TTransportException: Frame size larger
than protect max size is incorrect, you can adjust this parameter
appropriately.
[...]
+| -tp
| Specify time precision, optional values include 'ms' (milliseconds),' ns'
(nanoseconds), 'us' (microseconds)
[...]
+| -lpf
| Specify the number of lines to write data to each failed import file
[...]
+| -typeInfer
| Used to specify type inference rules. For
Example:<srcTsDataType1=dstTsDataType1,srcTsDataType2=dstTsDataType2,...>.
<br/>Note: Used to specify type inference rules.`srcTsDataType` include
`boolean`,`int`,`long`,`float`,`double`,`NaN`.`dstTsDataType` include
`boolean`,`int`,`long`,`float`,`double`,`text`.when`srcTsDataType`is`boolean`,
`dstTsDataType`can only be`boolean`or`text`.when`srcTsDataType`is`NaN`, ` [...]
+
+## Running Examples
+
+- Import the `dump0_0.sql` data in the current data directory into the local
IoTDB database.
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -s ./data/dump0_0.sql
+# Windows
+>tools/import-data.bat -s ./data/dump0_0.sql
+```
+
+- Import all data in the current data directory in an aligned manner into the
local IoTDB database.
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -s ./data/ -fd ./failed/ -aligned true
+# Windows
+>tools/import-data.bat -s ./data/ -fd ./failed/ -aligned true
+```
+
+- Import the `dump0_0.csv` data in the current data directory into the local
IoTDB database.
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -s ./data/dump0_0.csv -fd ./failed/
+# Windows
+>tools/import-data.bat -s ./data/dump0_0.csv -fd ./failed/
+```
+
+- Import the `dump0_0.csv` data in the current data directory in an aligned
manner, batch import 100,000 records into the IoTDB database on the host with
IP `192.168.100.1`, record failures in the current `failed` directory, and
limit each file to 1,000 records.
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -h 192.168.100.1 -p 6667 -u root -pw root -s
./data/dump0_0.csv -fd ./failed/ -aligned true -batch 100000 -tp ms -typeInfer
boolean=text,float=double -lpf 1000
+# Windows
+>tools/import-data.bat -h 192.168.100.1 -p 6667 -u root -pw root -s
./data/dump0_0.csv -fd ./failed/ -aligned true -batch 100000 -tp ms -typeInfer
boolean=text,float=double -lpf 1000
+```
\ No newline at end of file
diff --git a/src/UserGuide/Master/Tools-System/Import-Export-Tool.md
b/src/UserGuide/Master/Tools-System/TsFile-Import-Export-Tool.md
similarity index 99%
rename from src/UserGuide/Master/Tools-System/Import-Export-Tool.md
rename to src/UserGuide/Master/Tools-System/TsFile-Import-Export-Tool.md
index 47a4684..952a1db 100644
--- a/src/UserGuide/Master/Tools-System/Import-Export-Tool.md
+++ b/src/UserGuide/Master/Tools-System/TsFile-Import-Export-Tool.md
@@ -19,11 +19,11 @@
-->
-# Import Export Tool
+# TsFile Import Export Script
For different scenarios, IoTDB provides users with a variety of operation
methods for batch importing data. This chapter introduces the two most commonly
used methods for importing in the form of CSV text and importing in the form of
TsFile files.
-## TsFile Load And Export Tool
+## TsFile Load And Export Script
### TsFile Load Tool
diff --git a/src/UserGuide/Master/User-Manual/Write-Delete-Data.md
b/src/UserGuide/Master/User-Manual/Write-Delete-Data.md
index cf23144..f0285db 100644
--- a/src/UserGuide/Master/User-Manual/Write-Delete-Data.md
+++ b/src/UserGuide/Master/User-Manual/Write-Delete-Data.md
@@ -185,11 +185,11 @@ In different scenarios, the IoTDB provides a variety of
methods for importing da
### TsFile Batch Load
-TsFile is the file format of time series used in IoTDB. You can directly
import one or more TsFile files with time series into another running IoTDB
instance through tools such as CLI. For details, see
[Import-Export-Tool](../Tools-System/Import-Export-Tool.md).
+TsFile is the file format of time series used in IoTDB. You can directly
import one or more TsFile files with time series into another running IoTDB
instance through tools such as CLI. For details, see
[Import-Export-Tool](../Tools-System/TsFile-Import-Export-Tool.md).
### CSV Batch Load
-CSV stores table data in plain text. You can write multiple formatted data
into a CSV file and import the data into the IoTDB in batches. Before importing
data, you are advised to create the corresponding metadata in the IoTDB. Don't
worry if you forget to create one, the IoTDB can automatically infer the data
in the CSV to its corresponding data type, as long as you have a unique data
type for each column. In addition to a single file, the tool supports importing
multiple CSV files as f [...]
+CSV stores table data in plain text. You can write multiple formatted data
into a CSV file and import the data into the IoTDB in batches. Before importing
data, you are advised to create the corresponding metadata in the IoTDB. Don't
worry if you forget to create one, the IoTDB can automatically infer the data
in the CSV to its corresponding data type, as long as you have a unique data
type for each column. In addition to a single file, the tool supports importing
multiple CSV files as f [...]
## DELETE
diff --git a/src/UserGuide/latest/SQL-Manual/SQL-Manual.md
b/src/UserGuide/latest/SQL-Manual/SQL-Manual.md
index ca0ffff..a943ff2 100644
--- a/src/UserGuide/latest/SQL-Manual/SQL-Manual.md
+++ b/src/UserGuide/latest/SQL-Manual/SQL-Manual.md
@@ -442,7 +442,7 @@ IoTDB > select * from root.sg1.d1
### Load External TsFile Tool
-For more details, see document
[Import-Export-Tool](../Tools-System/Import-Export-Tool.md).
+For more details, see document
[Import-Export-Tool](../Tools-System/TsFile-Import-Export-Tool.md).
#### Load with SQL
diff --git a/src/UserGuide/latest/Tools-System/Data-Import-Export-Tool.md
b/src/UserGuide/latest/Tools-System/Data-Import-Export-Tool.md
new file mode 100644
index 0000000..f988a77
--- /dev/null
+++ b/src/UserGuide/latest/Tools-System/Data-Import-Export-Tool.md
@@ -0,0 +1,278 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# Data Import Export Script
+
+IoTDB provides data import and export scripts (tools/export-data,
tools/import-data, supported in versions 1.3.2 and above; for historical
versions, tools/export-csv, tools/import-csv scripts can be used, see the
reference link for usage [Import Export Tool](./TsFile-Import-Export-Tool.md)
), which are used to facilitate the interaction between IoTDB internal data and
external files, suitable for batch operations of single files or directories.
+
+
+## Supported Data Formats
+
+- **CSV** : Plain text format for storing formatted data, which must be
constructed according to the specified CSV format below.
+- **SQL** : Files containing custom SQL statements.
+
+# export-data Script (Data Export)
+
+## Command
+
+```Bash
+# Unix/OS X
+>tools/export-data.sh -h <ip> -p <port> -u <username> -pw <password> -t
<directory> [-tf <time-format> -datatype <true/false> -q <query command> -s
<source sql file> -tfn <target file name> -lpf <int> -type <export type>
-aligned <true/false>]
+
+# Windows
+>tools\export-data.bat -h <ip> -p <port> -u <username> -pw <password> -t
<directory> [-tf <time-format> -datatype <true/false> -q <query command> -s
<source sql file> -tfn <target file name> -lpf <int> -type <export type>
-aligned <true/false>]
+```
+
+Parameter Introduction:
+
+| Parameter | Definition
| Required | Default
|
+|:-------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------|:-------------------------|
+| -h | Database IP address
| No | 127.0.0.1
|
+| -p | Database port
| No | 6667
|
+| -u | Database connection username
| No | root
|
+| -pw | Database connection password
| No | root
|
+| -t | Output path for the exported CSV or SQL file
| Yes |
|
+| -datatype | Whether to print the corresponding data type behind the time
series in the CSV file header, options are true or false
| No | true
|
+| -q | Directly specify the query statement to be executed in the
command (currently only supports some statements, see the table below for
details). <br/>Note: -q and -s parameters must be filled in one, and -q takes
effect if both are filled. For detailed examples of supported SQL statements,
please refer to the "SQL Statement Support Details" below.
| No |
|
+| -s | Specify an SQL file, which may contain one or more SQL
statements. If there are multiple SQL statements, they should be separated by
newlines (returns). Each SQL statement corresponds to one or more output CSV or
SQL files. <br/>Note: -q and -s parameters must be filled in one, and -q takes
effect if both are filled. For detailed examples of supported SQL statements,
please refer to the "SQL Statement Support Details" below. | No |
|
+| -type | Specify the type of exported file, options are csv or sql
| No | csv
|
+| -tf | Specify the time format. The time format must comply with the
[ISO 8601](https://calendars.wikia.org/wiki/ISO_8601) standard or timestamp.
<br/>Note: Only effective when -type is csv
| No | yyyy-MM-dd
HH:mm:ss.SSSz |
+| -lpf | Specify the maximum number of lines for the exported dump
file
| No | 10000
|
+| -timeout | Specify the timeout time for session queries in milliseconds
| No | -1
|
+
+SQL Statement Support Rules:
+
+1. Only query statements are supported; non-query statements (such as metadata
management, system management, etc.) are not supported. For unsupported SQL,
the program will automatically skip and output an error message.
+2. In the current version of query statements, only the export of raw data is
supported. If there are group by, aggregate functions, UDFs, or operational
operators, they are not supported for export as SQL. When exporting raw data,
please note that if exporting data from multiple devices, please use the align
by device statement. Detailed examples are as follows:
+
+| | Supported for
Export | Example |
+|--------------------------------------------------------|----------------------|-----------------------------------------------|
+| Raw data single device query | Supported
| select * from root.s_0.d_0 |
+| Raw data multi-device query (align by device) | Supported
| select * from root.** align by device |
+| Raw data multi-device query (without align by device) | Unsupported
| select * from root.**<br/>select * from root.s_0.* |
+
+## Running Examples
+
+- Export all data within a certain SQL execution range to a CSV file.
+```Bash
+ # Unix/OS X
+ >tools/export-data.sh -t ./data/ -q 'select * from root.stock.**'
+ # Windows
+ >tools/export-data.bat -t ./data/ -q 'select * from root.stock.**'
+ ```
+
+- Export Results
+ ```Bash
+
Time,root.stock.Legacy.0700HK.L1_BidPrice,root.stock.Legacy.0700HK.Type,root.stock.Legacy.0700HK.L1_BidSize,root.stock.Legacy.0700HK.Domain,root.stock.Legacy.0700HK.L1_BuyNo,root.stock.Legacy.0700HK.L1_AskPrice
+
2024-07-29T18:37:18.700+08:00,0.9666617,3.0,0.021367407654674264,-6.0,false,0.8926191
+
2024-07-29T18:37:19.701+08:00,0.3057328,3.0,0.9965377284981661,-5.0,false,0.15167356
+ ```
+- All data within the scope of all SQL statements in the SQL file is exported
to CSV files.
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -t ./data/ -s export.sql
+ # Windows
+ >tools/export-data.bat -t ./data/ -s export.sql
+ ```
+
+- Contents of export.sql File (Pointed to by -s Parameter)
+ ```SQL
+ select * from root.stock.** limit 100
+ select * from root.db.** limit 100
+ ```
+
+- Export Result File 1
+ ```Bash
+
Time,root.stock.Legacy.0700HK.L1_BidPrice,root.stock.Legacy.0700HK.Type,root.stock.Legacy.0700HK.L1_BidSize,root.stock.Legacy.0700HK.Domain,root.stock.Legacy.0700HK.L1_BuyNo,root.stock.Legacy.0700HK.L1_AskPrice
+
2024-07-29T18:37:18.700+08:00,0.9666617,3.0,0.021367407654674264,-6.0,false,0.8926191
+
2024-07-29T18:37:19.701+08:00,0.3057328,3.0,0.9965377284981661,-5.0,false,0.15167356
+ ```
+
+- Export Result File 2
+ ```Bash
+ Time,root.db.Random.RandomBoolean
+ 2024-07-22T17:16:05.820+08:00,true
+ 2024-07-22T17:16:02.597+08:00,false
+ ```
+- Export Data in SQL File to SQL Statements with Aligned Format
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -h 127.0.0.1 -p 6667 -u root -p root -t ./data/ -s
export.sql -type sql -aligned true
+ # Windows
+ >tools/export-data.bat -h 127.0.0.1 -p 6667 -u root -p root -t ./data/ -s
export.sql -type sql -aligned true
+ ```
+
+- Export Results
+ ```Bash
+ INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
ALIGNED VALUES
(1722249629831,0.62308747,2.0,0.012206747854849653,-6.0,false,0.14164352);
+ INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
ALIGNED VALUES
(1722249630834,0.7520042,3.0,0.22760657101910464,-5.0,true,0.089064896);
+ INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
ALIGNED VALUES
(1722249631835,0.3981064,3.0,0.6254559288663467,-6.0,false,0.9767922);
+ ```
+- Export Data in a Certain SQL Execution Range to a CSV File with Specified
Time Format and Data Types
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -t ./data/ -tf 'yyyy-MM-dd HH:mm:ss' -datatype true -q
"select * from root.stock.**" -type csv
+ # Windows
+ >tools/export-data.bat -t ./data/ -tf 'yyyy-MM-dd HH:mm:ss' -datatype true
-q "select * from root.stock.**" -type csv
+ ```
+
+- Export Results
+ ```Bash
+
Time,root.stock.Legacy.0700HK.L1_BidPrice(DOUBLE),root.stock.Legacy.0700HK.Type(DOUBLE),root.stock.Legacy.0700HK.L1_BidSize(DOUBLE),root.stock.Legacy.0700HK.Domain(DOUBLE),root.stock.Legacy.0700HK.L1_BuyNo(BOOLEAN),root.stock.Legacy.0700HK.L1_AskPrice(DOUBLE)
+ 2024-07-30 10:33:55,0.44574088,3.0,0.21476832811611501,-4.0,true,0.5951748
+ 2024-07-30 10:33:56,0.6880933,3.0,0.6289119476165305,-5.0,false,0.114634395
+ ```
+
+# import-data Script (Data Import)
+
+## Import File Examples
+
+### CSV File Example
+
+Note that before importing CSV data, special characters need to be handled as
follows:
+
+1. If the text type field contains special characters such as `,`, it should
be escaped with `\`.
+2. You can import times in formats like `yyyy-MM-dd'T'HH:mm:ss`, `yyyy-MM-dd
HH:mm:ss`, or `yyyy-MM-dd'T'HH:mm:ss.SSSZ`.
+3. The time column `Time` should always be in the first column.
+
+Example 1: Time Aligned, No Data Types in Header
+
+```SQL
+Time,root.test.t1.str,root.test.t2.str,root.test.t2.var
+1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
+1970-01-01T08:00:00.002+08:00,"123",,
+```
+
+Example 2: Time Aligned, Data Types in Header(Text type data supports double
quotation marks and non double quotation marks)
+
+```SQL
+Time,root.test.t1.str(TEXT),root.test.t2.str(TEXT),root.test.t2.var(INT32)
+1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
+1970-01-01T08:00:00.002+08:00,123,hello world,123
+1970-01-01T08:00:00.003+08:00,"123",,
+1970-01-01T08:00:00.004+08:00,123,,12
+```
+Example 3: Device Aligned, No Data Types in Header
+
+```SQL
+Time,Device,str,var
+1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
+1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
+1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
+```
+
+Example 4: Device Aligned, Data Types in Header (Text type data supports
double quotation marks and non double quotation marks)
+
+```SQL
+Time,Device,str(TEXT),var(INT32)
+1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
+1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
+1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
+1970-01-01T08:00:00.002+08:00,root.test.t1,hello world,123
+```
+
+### SQL File Example
+
+> For unsupported SQL, illegal SQL, or failed SQL executions, they will be
placed in the failed directory under the failed file (default to
filename.failed).
+
+```SQL
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728578812,0.21911979,4.0,0.7129878488375604,-5.0,false,0.65362453);
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728579812,0.35814416,3.0,0.04674720094979623,-5.0,false,0.9365247);
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728580813,0.20012152,3.0,0.9910098187911393,-4.0,true,0.70040536);
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728581814,0.034122765,4.0,0.9313345284181858,-4.0,true,0.9945297);
+```
+
+## Command
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -h <ip> -p <port> -u <username> -pw <password> -s
<xxx.csv/sql> [-fd <./failedDirectory> -aligned <true/false> -batch <int> -tp
<ms/ns/us> -typeInfer <boolean=text,float=double...> -lpf <int>]
+
+# Windows
+>tools\import-data.bat -h <ip> -p <port> -u <username> -pw <password> -s
<xxx.csv/sql> [-fd <./failedDirectory> -aligned <true/false> -batch <int> -tp
<ms/ns/us> -typeInfer <boolean=text,float=double...> -lpf <int>]
+```
+
+> Although IoTDB has the ability to infer types, it is still recommended to
create metadata before importing data to avoid unnecessary type conversion
errors. For example:
+
+```SQL
+CREATE DATABASE root.fit.d1;
+CREATE DATABASE root.fit.d2;
+CREATE DATABASE root.fit.p;
+CREATE TIMESERIES root.fit.d1.s1 WITH DATATYPE=INT32,ENCODING=RLE;
+CREATE TIMESERIES root.fit.d1.s2 WITH DATATYPE=TEXT,ENCODING=PLAIN;
+CREATE TIMESERIES root.fit.d2.s1 WITH DATATYPE=INT32,ENCODING=RLE;
+CREATE TIMESERIES root.fit.d2.s3 WITH DATATYPE=INT32,ENCODING=RLE;
+CREATE TIMESERIES root.fit.p.s1 WITH DATATYPE=INT32,ENCODING=RLE;
+```
+
+Parameter Introduction:
+
+| Parameter
| Definition
[...]
+|:----------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| -h
| Database IP address
[...]
+| -p
| Database port
[...]
+| -u
| Database connection username
[...]
+| -pw
| Database connection password
[...]
+| -s
| Specify the data you want to import, here you can specify a file or folder.
If a folder is specified, all files with the suffix CSV or SQL in the folder
will be imported in bulk.
[...]
+| -fd
| Specify the directory to store the failed SQL files. If this parameter is not
specified, the failed files will be saved to the directory of the source data.
<br/>Note: For unsupported SQL, illegal SQL, and failed SQL, they will be
placed in the failed file in the failed directory (default file name is.
failed)
[...]
+| -aligned
| Specify whether to use the 'aligned' interface, with options of true or
false. This parameter only takes effect when the imported file is a CSV file
[...]
+| -batch
| Used to specify the number of points to be inserted for each batch of data
(minimum value is 1, maximum value is Integer.MAX_VALUE). If the program
reports' org.apache.hrift.transport ' If TTransportException: Frame size larger
than protect max size is incorrect, you can adjust this parameter
appropriately.
[...]
+| -tp
| Specify time precision, optional values include 'ms' (milliseconds),' ns'
(nanoseconds), 'us' (microseconds)
[...]
+| -lpf
| Specify the number of lines to write data to each failed import file
[...]
+| -typeInfer
| Used to specify type inference rules. For
Example:<srcTsDataType1=dstTsDataType1,srcTsDataType2=dstTsDataType2,...>.<br/>Note:
Used to specify type inference rules.`srcTsDataType` include
`boolean`,`int`,`long`,`float`,`double`,`NaN`.`dstTsDataType` include
`boolean`,`int`,`long`,`float`,`double`,`text`.when`srcTsDataType`is`boolean`,
`dstTsDataType`can only be`boolean`or`text`.when`srcTsDataType`is`NaN`, `ds
[...]
+
+## Running Examples
+
+- Import the `dump0_0.sql` data in the current data directory into the local
IoTDB database.
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -s ./data/dump0_0.sql
+# Windows
+>tools/import-data.bat -s ./data/dump0_0.sql
+```
+
+- Import all data in the current data directory in an aligned manner into the
local IoTDB database.
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -s ./data/ -fd ./failed/ -aligned true
+# Windows
+>tools/import-data.bat -s ./data/ -fd ./failed/ -aligned true
+```
+
+- Import the `dump0_0.csv` data in the current data directory into the local
IoTDB database.
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -s ./data/dump0_0.csv -fd ./failed/
+# Windows
+>tools/import-data.bat -s ./data/dump0_0.csv -fd ./failed/
+```
+
+- Import the `dump0_0.csv` data in the current data directory in an aligned
manner, batch import 100,000 records into the IoTDB database on the host with
IP `192.168.100.1`, record failures in the current `failed` directory, and
limit each file to 1,000 records.
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -h 192.168.100.1 -p 6667 -u root -pw root -s
./data/dump0_0.csv -fd ./failed/ -aligned true -batch 100000 -tp ms -typeInfer
boolean=text,float=double -lpf 1000
+# Windows
+>tools/import-data.bat -h 192.168.100.1 -p 6667 -u root -pw root -s
./data/dump0_0.csv -fd ./failed/ -aligned true -batch 100000 -tp ms -typeInfer
boolean=text,float=double -lpf 1000
+```
\ No newline at end of file
diff --git a/src/UserGuide/latest/Tools-System/Import-Export-Tool.md
b/src/UserGuide/latest/Tools-System/TsFile-Import-Export-Tool.md
similarity index 99%
rename from src/UserGuide/latest/Tools-System/Import-Export-Tool.md
rename to src/UserGuide/latest/Tools-System/TsFile-Import-Export-Tool.md
index 47a4684..952a1db 100644
--- a/src/UserGuide/latest/Tools-System/Import-Export-Tool.md
+++ b/src/UserGuide/latest/Tools-System/TsFile-Import-Export-Tool.md
@@ -19,11 +19,11 @@
-->
-# Import Export Tool
+# TsFile Import Export Script
For different scenarios, IoTDB provides users with a variety of operation
methods for batch importing data. This chapter introduces the two most commonly
used methods for importing in the form of CSV text and importing in the form of
TsFile files.
-## TsFile Load And Export Tool
+## TsFile Load And Export Script
### TsFile Load Tool
diff --git a/src/UserGuide/latest/User-Manual/Write-Delete-Data.md
b/src/UserGuide/latest/User-Manual/Write-Delete-Data.md
index cf23144..f0285db 100644
--- a/src/UserGuide/latest/User-Manual/Write-Delete-Data.md
+++ b/src/UserGuide/latest/User-Manual/Write-Delete-Data.md
@@ -185,11 +185,11 @@ In different scenarios, the IoTDB provides a variety of
methods for importing da
### TsFile Batch Load
-TsFile is the file format of time series used in IoTDB. You can directly
import one or more TsFile files with time series into another running IoTDB
instance through tools such as CLI. For details, see
[Import-Export-Tool](../Tools-System/Import-Export-Tool.md).
+TsFile is the file format of time series used in IoTDB. You can directly
import one or more TsFile files with time series into another running IoTDB
instance through tools such as CLI. For details, see
[Import-Export-Tool](../Tools-System/TsFile-Import-Export-Tool.md).
### CSV Batch Load
-CSV stores table data in plain text. You can write multiple formatted data
into a CSV file and import the data into the IoTDB in batches. Before importing
data, you are advised to create the corresponding metadata in the IoTDB. Don't
worry if you forget to create one, the IoTDB can automatically infer the data
in the CSV to its corresponding data type, as long as you have a unique data
type for each column. In addition to a single file, the tool supports importing
multiple CSV files as f [...]
+CSV stores table data in plain text. You can write multiple formatted data
into a CSV file and import the data into the IoTDB in batches. Before importing
data, you are advised to create the corresponding metadata in the IoTDB. Don't
worry if you forget to create one, the IoTDB can automatically infer the data
in the CSV to its corresponding data type, as long as you have a unique data
type for each column. In addition to a single file, the tool supports importing
multiple CSV files as f [...]
## DELETE
diff --git a/src/zh/Download/README.md b/src/zh/Download/README.md
index 11898c5..7604c9b 100644
--- a/src/zh/Download/README.md
+++ b/src/zh/Download/README.md
@@ -219,7 +219,7 @@ insert into root.sg(time, a) values(4, "\u96d5")
* 停掉旧版本进程,启动新版本即可。
- 如何从 v0.13.x 升级到 v1.0.x?
- * 0.13版本与1.0版本的文件目录结构不同,**<font
color=red>不能将0.13的data目录直接拷贝到1.0集群使用</font>**。如果需要将0.13的数据导入至1.0,可以使用[LOAD](../UserGuide/Master/Tools-System/Import-Export-Tool.md)功能;
+ * 0.13版本与1.0版本的文件目录结构不同,**<font
color=red>不能将0.13的data目录直接拷贝到1.0集群使用</font>**。如果需要将0.13的数据导入至1.0,可以使用[LOAD](../UserGuide/Master/Tools-System/TsFile-Import-Export-Tool.md)功能;
* __0.13版本的默认RPC地址是0.0.0.0,1.0版本的默认RPC地址是127.0.0.1__
- 如何从 v1.0.0 升级到 v1.x.0?
diff --git a/src/zh/UserGuide/Master/Tools-System/Data-Import-Export-Tool.md
b/src/zh/UserGuide/Master/Tools-System/Data-Import-Export-Tool.md
new file mode 100644
index 0000000..30111de
--- /dev/null
+++ b/src/zh/UserGuide/Master/Tools-System/Data-Import-Export-Tool.md
@@ -0,0 +1,278 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# 数据导入导出脚本
+
+IoTDB 提供了数据导入导出脚本(tools/export-data、tools/import-data,V1.3.2 及之后版本支持;历史版本可使用
tools/export-csv、tools/import-csv
脚本,使用参考[导入导出工具](./TsFile-Import-Export-Tool.md)),用于实现 IoTDB
内部数据与外部文件的交互,适用于单个文件或目录文件批量操作。
+
+## 支持的数据格式
+
+- **CSV**:纯文本格式,存储格式化数据,需按照下文指定 CSV 格式进行构造
+- **SQL**:包含自定义 SQL 语句的文件
+
+# export-data 脚本(数据导出)
+
+## 运行命令
+
+```Bash
+# Unix/OS X
+>tools/export-data.sh -h <ip> -p <port> -u <username> -pw <password> -t
<directory> [-tf <time-format> -datatype <true/false> -q <query command> -s
<source sql file> -tfn <target file name> -lpf <int> -type <export type>
-aligned <true/false>]
+
+# Windows
+>tools\export-data.bat -h <ip> -p <port> -u <username> -pw <password> -t
<directory> [-tf <time-format> -datatype <true/false> -q <query command> -s
<source sql file> -tfn <target file name> -lpf <int> -type <export type>
-aligned <true/false>]
+```
+
+参数介绍:
+
+| 参数 | 定义
| 是否必填 | 默认 |
+| :--------
|:-------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------|
:----------------------- |
+| -h | 数据库IP地址
| 否 | 127.0.0.1 |
+| -p | 数据库端口
| 否 | 6667 |
+| -u | 数据库连接用户名
| 否 | root |
+| -pw | 数据库连接密码
| 否 | root |
+| -t | 导出的 CSV 或 SQL 文件的输出路径
| 是 | |
+| -datatype | 是否在 CSV 文件的 header 中时间序列的后面打印出对应的数据类型,选项为 true 或者 false
| 否 | true |
+| -q | 在命令中直接指定想要执行的查询语句(目前仅支持部分语句,详细明细见下表)<br/>说明:-q 与 -s 参数必填其一,同时填写则
-q 生效。详细支持的 SQL 语句示例,请参考下方“SQL语句支持明细”
| 否 | |
+| -s | 指定 SQL 文件,该文件可包含一条或多条 SQL 语句。如果包含多条 SQL 语句,语句之间应该用换行(回车)进行分割。每一条
SQL 语句对应一个或多个输出的CSV或 SQL 文件<br/>说明:-q 与 -s 参数必填其一,同时填写则-q生效。详细支持的 SQL
语句示例,请参考下方“SQL语句支持规则” | 否 | |
+| -type | 指定导出的文件类型,选项为 csv 或者 sql
| 否 | csv |
+| -tf | 指定时间格式。时间格式必须遵守[ISO
8601](https://calendars.wikia.org/wiki/ISO_8601)标准,或时间戳(`timestamp`)<br/>说明:只在
-type 为 csv 时生效 | 否 |
yyyy-MM-dd HH:mm:ss.SSSz |
+| -lpf | 指定导出的 dump 文件最大行数
| 否 | 10000 |
+| -timeout | 指定 session 查询时的超时时间,单位为ms
| 否 | -1 |
+
+ SQL 语句支持规则:
+
+1. 只支持查询语句,非查询语句(如:元数据管理、系统管理等语句)不支持。对于不支持的 SQL ,程序会自动跳过,同时输出错误信息。
+2. 查询语句中目前版本仅支持原始数据的导出,如果有使用 group by、聚合函数、udf、操作运算符等则不支持导出为
SQL。原始数据导出时请注意,若导出多个设备数据,请使用 align by device 语句。详细示例如下:
+
+| | 支持导出 | 示例
|
+|-------------------------------| -------- |
--------------------------------------------- |
+| 原始数据单设备查询 | 支持 | select * from root.s_0.d_0
|
+| 原始数据多设备查询(aligin by device) | 支持 | select * from root.** align by
device |
+| 原始数据多设备查询(无 aligin by device) | 不支持 | select * from root.**<br/>select *
from root.s_0.* |
+
+## 运行示例
+
+- 导出某 SQL 执行范围下的所有数据至 CSV 文件。
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -t ./data/ -q 'select * from root.stock.**'
+ # Windows
+ >tools/export-data.bat -t ./data/ -q 'select * from root.stock.**'
+ ```
+
+- 导出结果
+ ```Bash
+
Time,root.stock.Legacy.0700HK.L1_BidPrice,root.stock.Legacy.0700HK.Type,root.stock.Legacy.0700HK.L1_BidSize,root.stock.Legacy.0700HK.Domain,root.stock.Legacy.0700HK.L1_BuyNo,root.stock.Legacy.0700HK.L1_AskPrice
+
2024-07-29T18:37:18.700+08:00,0.9666617,3.0,0.021367407654674264,-6.0,false,0.8926191
+
2024-07-29T18:37:19.701+08:00,0.3057328,3.0,0.9965377284981661,-5.0,false,0.15167356
+ ```
+- 导出 SQL 文件内所有 SQL 执行范围下的所有数据至 CSV 文件。
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -t ./data/ -s export.sql
+ # Windows
+ >tools/export-data.bat -t ./data/ -s export.sql
+ ```
+
+- export.sql 文件内容(-s 参数指向的文件)
+ ```SQL
+ select * from root.stock.** limit 100
+ select * from root.db.** limit 100
+ ```
+
+- 导出结果文件1
+ ```Bash
+
Time,root.stock.Legacy.0700HK.L1_BidPrice,root.stock.Legacy.0700HK.Type,root.stock.Legacy.0700HK.L1_BidSize,root.stock.Legacy.0700HK.Domain,root.stock.Legacy.0700HK.L1_BuyNo,root.stock.Legacy.0700HK.L1_AskPrice
+
2024-07-29T18:37:18.700+08:00,0.9666617,3.0,0.021367407654674264,-6.0,false,0.8926191
+
2024-07-29T18:37:19.701+08:00,0.3057328,3.0,0.9965377284981661,-5.0,false,0.15167356
+ ```
+
+- 导出结果文件2
+ ```Bash
+ Time,root.db.Random.RandomBoolean
+ 2024-07-22T17:16:05.820+08:00,true
+ 2024-07-22T17:16:02.597+08:00,false
+ ```
+- 将 IoTDB 数据库中在 SQL 文件内定义的数据,以对齐的格式将其导出为 SQL 语句。
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -h 127.0.0.1 -p 6667 -u root -p root -t ./data/ -s
export.sql -type sql -aligned true
+ # Windows
+ >tools/export-data.bat -h 127.0.0.1 -p 6667 -u root -p root -t ./data/ -s
export.sql -type sql -aligned true
+ ```
+
+- 导出结果
+ ```Bash
+ INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
ALIGNED VALUES
(1722249629831,0.62308747,2.0,0.012206747854849653,-6.0,false,0.14164352);
+ INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
ALIGNED VALUES
(1722249630834,0.7520042,3.0,0.22760657101910464,-5.0,true,0.089064896);
+ INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
ALIGNED VALUES
(1722249631835,0.3981064,3.0,0.6254559288663467,-6.0,false,0.9767922);
+ ```
+- 将某 SQL 执行范围下的所有数据导出至 CSV 文件,指定导出的时间格式为`yyyy-MM-dd
HH:mm:ss`,且表头时间序列的后面打印出对应的数据类型。
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -t ./data/ -tf 'yyyy-MM-dd HH:mm:ss' -datatype true
-q "select * from root.stock.**" -type csv
+ # Windows
+ >tools/export-data.bat -t ./data/ -tf 'yyyy-MM-dd HH:mm:ss' -datatype true
-q "select * from root.stock.**" -type csv
+ ```
+
+- 导出结果
+ ```Bash
+
Time,root.stock.Legacy.0700HK.L1_BidPrice(DOUBLE),root.stock.Legacy.0700HK.Type(DOUBLE),root.stock.Legacy.0700HK.L1_BidSize(DOUBLE),root.stock.Legacy.0700HK.Domain(DOUBLE),root.stock.Legacy.0700HK.L1_BuyNo(BOOLEAN),root.stock.Legacy.0700HK.L1_AskPrice(DOUBLE)
+ 2024-07-30 10:33:55,0.44574088,3.0,0.21476832811611501,-4.0,true,0.5951748
+ 2024-07-30 10:33:56,0.6880933,3.0,0.6289119476165305,-5.0,false,0.114634395
+ ```
+
+# import-data 脚本(数据导入)
+
+## 导入文件示例
+
+### CSV 文件示例
+
+注意,在导入 CSV 数据前,需要特殊处理下列的字符:
+
+1. 如果 Text 类型的字段中包含特殊字符如`,`需要使用`\`来进行转义。
+2. 可以导入像`yyyy-MM-dd'T'HH:mm:ss`, `yyy-MM-dd HH:mm:ss`, 或者
`yyyy-MM-dd'T'HH:mm:ss.SSSZ`格式的时间。
+3. 时间列`Time`应该始终放在第一列。
+
+示例一:时间对齐,并且 header 中不包含数据类型的数据。
+
+```SQL
+Time,root.test.t1.str,root.test.t2.str,root.test.t2.var
+1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
+1970-01-01T08:00:00.002+08:00,"123",,
+```
+
+示例二:时间对齐,并且 header 中包含数据类型的数据(Text 类型数据支持加双引号和不加双引号)
+
+```SQL
+Time,root.test.t1.str(TEXT),root.test.t2.str(TEXT),root.test.t2.var(INT32)
+1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
+1970-01-01T08:00:00.002+08:00,123,hello world,123
+1970-01-01T08:00:00.003+08:00,"123",,
+1970-01-01T08:00:00.004+08:00,123,,12
+```
+
+示例三:设备对齐,并且 header 中不包含数据类型的数据
+
+```SQL
+Time,Device,str,var
+1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
+1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
+1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
+```
+
+示例四:设备对齐,并且 header 中包含数据类型的数据(Text 类型数据支持加双引号和不加双引号)
+
+```SQL
+Time,Device,str(TEXT),var(INT32)
+1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
+1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
+1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
+1970-01-01T08:00:00.002+08:00,root.test.t1,hello world,123
+```
+
+### SQL 文件示例
+
+> 对于不支持的 SQL ,不合法的 SQL ,执行失败的 SQL 都会放到失败目录下的失败文件里(默认为 文件名.failed)
+
+```SQL
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728578812,0.21911979,4.0,0.7129878488375604,-5.0,false,0.65362453);
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728579812,0.35814416,3.0,0.04674720094979623,-5.0,false,0.9365247);
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728580813,0.20012152,3.0,0.9910098187911393,-4.0,true,0.70040536);
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728581814,0.034122765,4.0,0.9313345284181858,-4.0,true,0.9945297);
+```
+
+## 运行命令
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -h <ip> -p <port> -u <username> -pw <password> -s
<xxx.csv/sql> [-fd <./failedDirectory> -aligned <true/false> -batch <int> -tp
<ms/ns/us> -typeInfer <boolean=text,float=double...> -lpf <int>]
+
+# Windows
+>tools\import-data.bat -h <ip> -p <port> -u <username> -pw <password> -s
<xxx.csv/sql> [-fd <./failedDirectory> -aligned <true/false> -batch <int> -tp
<ms/ns/us> -typeInfer <boolean=text,float=double...> -lpf <int>]
+```
+
+> 虽然 IoTDB 具有类型推断的能力,但我们仍然推荐在导入数据前创建元数据,因为这可以避免不必要的类型转换错误。如下:
+
+```SQL
+CREATE DATABASE root.fit.d1;
+CREATE DATABASE root.fit.d2;
+CREATE DATABASE root.fit.p;
+CREATE TIMESERIES root.fit.d1.s1 WITH DATATYPE=INT32,ENCODING=RLE;
+CREATE TIMESERIES root.fit.d1.s2 WITH DATATYPE=TEXT,ENCODING=PLAIN;
+CREATE TIMESERIES root.fit.d2.s1 WITH DATATYPE=INT32,ENCODING=RLE;
+CREATE TIMESERIES root.fit.d2.s3 WITH DATATYPE=INT32,ENCODING=RLE;
+CREATE TIMESERIES root.fit.p.s1 WITH DATATYPE=INT32,ENCODING=RLE;
+```
+
+参数介绍:
+
+| 参数 | 定义
| 是否必填 | 默认 [...]
+|:-------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
:------- | : [...]
+| -h | 数据库 IP 地址
| 否 | 12 [...]
+| -p | 数据库端口
| 否 | 66 [...]
+| -u | 数据库连接用户名
| 否 | ro [...]
+| -pw | 数据库连接密码
| 否 | ro [...]
+| -s | 指定想要导入的数据,这里可以指定文件或者文件夹。如果指定的是文件夹,将会把文件夹中所有的后缀为 csv 或者 sql
的文件进行批量导入。
| 是 | [...]
+| -fd | 指定存放失败 SQL 文件的目录,如果未指定这个参数,失败的文件将会被保存到源数据的目录中。<br/>说明:对于不支持的
SQL ,不合法的 SQL ,执行失败的 SQL 都会放到失败目录下的失败文件里(默认为 文件名.failed)
| 否 | 源文 [...]
+| -aligned | 指定是否使用`aligned`接口,选项为 true 或者 false
<br/>说明:这个参数只在导入文件为csv文件时生效
| 否 | fa [...]
+| -batch | 用于指定每一批插入的数据的点数(最小值为1,最大值为
Integer.*MAX_VALUE*)。如果程序报了`org.apache.thrift.transport.TTransportException:
Frame size larger than protect max size`这个错的话,就可以适当的调低这个参数。
| 否 | `1 [...]
+| -tp | 指定时间精度,可选值包括`ms`(毫秒),`ns`(纳秒),`us`(微秒)
| 否 | `m [...]
+| -lpf | 指定每个导入失败文件写入数据的行数
| 否 | 10 [...]
+| -typeInfer |
用于指定类型推断规则,如<srcTsDataType1=dstTsDataType1,srcTsDataType2=dstTsDataType2,...>。<br/>说明:用于指定类型推断规则.`srcTsDataType`
包括 `boolean`,`int`,`long`,`float`,`double`,`NaN`.`dstTsDataType` 包括
`boolean`,`int`,`long`,`float`,`double`,`text`.当`srcTsDataType`为`boolean`,
`dstTsDataType`只能为`boolean`或`text`.当`srcTsDataType`为`NaN`,
`dstTsDataType`只能为`float`, `double`或`text`.当`srcTsDataType`为数值类型,
`dstTsDataType`的精度需要高于`srcTsDataType`.例如:`-typeInfer boolean=text,float=double`
| 否 | [...]
+
+## 运行示例
+
+- 导入当前`data`目录下的`dump0_0.sql`数据到本机 IoTDB 数据库中。
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -s ./data/dump0_0.sql
+# Windows
+>tools/import-data.bat -s ./data/dump0_0.sql
+```
+
+- 将当前`data`目录下的所有数据以对齐的方式导入到本机 IoTDB 数据库中。
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -s ./data/ -fd ./failed/ -aligned true
+# Windows
+>tools/import-data.bat -s ./data/ -fd ./failed/ -aligned true
+```
+
+- 导入当前`data`目录下的`dump0_0.csv`数据到本机 IoTDB 数据库中。
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -s ./data/dump0_0.csv -fd ./failed/
+# Windows
+>tools/import-data.bat -s ./data/dump0_0.csv -fd ./failed/
+```
+
+- 将当前`data`目录下的`dump0_0.csv`数据以对齐的方式,一批导入100000条导入到`192.168.100.1`IP所在主机的
IoTDB 数据库中,失败的记录记在当前`failed`目录下,每个文件最多记1000条。
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -h 192.168.100.1 -p 6667 -u root -pw root -s
./data/dump0_0.csv -fd ./failed/ -aligned true -batch 100000 -tp ms -typeInfer
boolean=text,float=double -lpf 1000
+# Windows
+>tools/import-data.bat -h 192.168.100.1 -p 6667 -u root -pw root -s
./data/dump0_0.csv -fd ./failed/ -aligned true -batch 100000 -tp ms -typeInfer
boolean=text,float=double -lpf 1000
+```
\ No newline at end of file
diff --git a/src/zh/UserGuide/Master/Tools-System/Import-Export-Tool.md
b/src/zh/UserGuide/Master/Tools-System/TsFile-Import-Export-Tool.md
similarity index 99%
rename from src/zh/UserGuide/Master/Tools-System/Import-Export-Tool.md
rename to src/zh/UserGuide/Master/Tools-System/TsFile-Import-Export-Tool.md
index 28b0b98..3e83baf 100644
--- a/src/zh/UserGuide/Master/Tools-System/Import-Export-Tool.md
+++ b/src/zh/UserGuide/Master/Tools-System/TsFile-Import-Export-Tool.md
@@ -19,17 +19,17 @@
-->
-# 导入导出工具
+# TsFile 导入导出脚本
针对于不同场景,IoTDB 为用户提供多种批量导入数据的操作方式,本章节向大家介绍最为常用的两种方式为 CSV文本形式的导入 和 TsFile文件形式的导入。
-## TsFile导入导出工具
+## TsFile 导入导出脚本
TsFile 是在 IoTDB 中使用的时间序列的文件格式,您可以通过CLI等工具直接将存有时间序列的一个或多个 TsFile
文件导入到另外一个正在运行的IoTDB实例中。
### 介绍
加载外部 tsfile 文件工具允许用户向正在运行中的 Apache IoTDB 中加载 tsfile
文件。或者您也可以使用脚本的方式将tsfile加载进IoTDB。
-### 使用SQL加载
+### 使用 SQL 加载
用户通过 Cli 工具或 JDBC 向 Apache IoTDB 系统发送指定命令实现文件加载的功能。
#### 加载 tsfile 文件
diff --git a/src/zh/UserGuide/Master/User-Manual/Write-Delete-Data.md
b/src/zh/UserGuide/Master/User-Manual/Write-Delete-Data.md
index d3b7ada..0966d28 100644
--- a/src/zh/UserGuide/Master/User-Manual/Write-Delete-Data.md
+++ b/src/zh/UserGuide/Master/User-Manual/Write-Delete-Data.md
@@ -176,11 +176,11 @@ It costs 0.004s
### TsFile批量导入
-TsFile 是在 IoTDB 中使用的时间序列的文件格式,您可以通过CLI等工具直接将存有时间序列的一个或多个 TsFile
文件导入到另外一个正在运行的IoTDB实例中。具体操作方式请参考[导入导出工具](../Tools-System/Import-Export-Tool.md)。
+TsFile 是在 IoTDB 中使用的时间序列的文件格式,您可以通过CLI等工具直接将存有时间序列的一个或多个 TsFile
文件导入到另外一个正在运行的IoTDB实例中。具体操作方式请参考[导入导出工具](../Tools-System/TsFile-Import-Export-Tool.md)。
### CSV批量导入
-CSV 是以纯文本形式存储表格数据,您可以在CSV文件中写入多条格式化的数据,并批量的将这些数据导入到 IoTDB
中,在导入数据之前,建议在IoTDB中创建好对应的元数据信息。如果忘记创建元数据也不要担心,IoTDB
可以自动将CSV中数据推断为其对应的数据类型,前提是你每一列的数据类型必须唯一。除单个文件外,此工具还支持以文件夹的形式导入多个 CSV
文件,并且支持设置如时间精度等优化参数。具体操作方式请参考[导入导出工具](../Tools-System/Import-Export-Tool.md)。
+CSV 是以纯文本形式存储表格数据,您可以在CSV文件中写入多条格式化的数据,并批量的将这些数据导入到 IoTDB
中,在导入数据之前,建议在IoTDB中创建好对应的元数据信息。如果忘记创建元数据也不要担心,IoTDB
可以自动将CSV中数据推断为其对应的数据类型,前提是你每一列的数据类型必须唯一。除单个文件外,此工具还支持以文件夹的形式导入多个 CSV
文件,并且支持设置如时间精度等优化参数。具体操作方式请参考[导入导出工具](../Tools-System/Data-Import-Export-Tool.md)。
## 删除数据
diff --git a/src/zh/UserGuide/latest/Tools-System/Data-Import-Export-Tool.md
b/src/zh/UserGuide/latest/Tools-System/Data-Import-Export-Tool.md
new file mode 100644
index 0000000..2416414
--- /dev/null
+++ b/src/zh/UserGuide/latest/Tools-System/Data-Import-Export-Tool.md
@@ -0,0 +1,278 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# 数据导入导出脚本
+
+IoTDB 提供了数据导入导出脚本(tools/export-data、tools/import-data,V1.3.2
及之后版本支持;历史版本可使用tools/export-csv、tools/import-csv脚本,使用参考[导入导出工具](./TsFile-Import-Export-Tool.md)),用于实现
IoTDB 内部数据与外部文件的交互,适用于单个文件或目录文件批量操作。
+
+## 支持的数据格式
+
+- **CSV**:纯文本格式,存储格式化数据,需按照下文指定 CSV 格式进行构造
+- **SQL**:包含自定义 SQL 语句的文件
+
+# export-data 脚本(数据导出)
+
+## 运行命令
+
+```Bash
+# Unix/OS X
+>tools/export-data.sh -h <ip> -p <port> -u <username> -pw <password> -t
<directory> [-tf <time-format> -datatype <true/false> -q <query command> -s
<source sql file> -tfn <target file name> -lpf <int> -type <export type>
-aligned <true/false>]
+
+# Windows
+>tools\export-data.bat -h <ip> -p <port> -u <username> -pw <password> -t
<directory> [-tf <time-format> -datatype <true/false> -q <query command> -s
<source sql file> -tfn <target file name> -lpf <int> -type <export type>
-aligned <true/false>]
+```
+
+参数介绍:
+
+| 参数 | 定义
| 是否必填 | 默认 |
+| :--------
|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------|
:----------------------- |
+| -h | 数据库 IP 地址
| 否 | 127.0.0.1 |
+| -p | 数据库端口
| 否 | 6667 |
+| -u | 数据库连接用户名
| 否 | root |
+| -pw | 数据库连接密码
| 否 | root |
+| -t | 导出的 CSV 或 SQL 文件的输出路径
| 是 | |
+| -datatype | 是否在 CSV 文件的 header 中时间序列的后面打印出对应的数据类型,选项为 true 或者 false
| 否 | true |
+| -q | 在命令中直接指定想要执行的查询语句(目前仅支持部分语句,详细明细见下表)<br/>说明:-q 与 -s 参数必填其一,同时填写则
-q 生效。详细支持的 SQL 语句示例,请参考下方“ SQL 语句支持明细”
| 否 | |
+| -s | 指定 SQL 文件,该文件可包含一条或多条 SQL 语句。如果包含多条 SQL 语句,语句之间应该用换行(回车)进行分割。每一条
SQL 语句对应一个或多个输出的 CSV 或 SQL 文件<br/>说明:-q 与 -s 参数必填其一,同时填写则 -q 生效。详细支持的 SQL
语句示例,请参考下方“ SQL 语句支持规则” | 否 | |
+| -type | 指定导出的文件类型,选项为 csv 或者 sql
| 否 | csv |
+| -tf | 指定时间格式。时间格式必须遵守[ISO
8601](https://calendars.wikia.org/wiki/ISO_8601)标准,或时间戳(`timestamp`)<br/>说明:只在
-type 为 csv 时生效 | 否 |
yyyy-MM-dd HH:mm:ss.SSSz |
+| -lpf | 指定导出的 dump 文件最大行数
| 否 | 10000 |
+| -timeout | 指定 session 查询时的超时时间,单位为 ms
| 否 | -1 |
+
+SQL 语句支持规则:
+
+1. 只支持查询语句,非查询语句(如:元数据管理、系统管理等语句)不支持。对于不支持的 SQL,程序会自动跳过,同时输出错误信息。
+2. 查询语句中目前版本仅支持原始数据的导出,如果有使用 group by、聚合函数、udf、操作运算符等则不支持导出为
SQL。原始数据导出时请注意,若导出多个设备数据,请使用 align by device 语句。详细示例如下:
+
+| | 支持导出 | 示例
|
+|-------------------------------| -------- |
--------------------------------------------- |
+| 原始数据单设备查询 | 支持 | select * from root.s_0.d_0
|
+| 原始数据多设备查询(aligin by device) | 支持 | select * from root.** align by
device |
+| 原始数据多设备查询(无 aligin by device) | 不支持 | select * from root.**<br/>select *
from root.s_0.* |
+
+## 运行示例
+
+- 导出某 SQL 执行范围下的所有数据至 CSV 文件。
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -t ./data/ -q 'select * from root.stock.**'
+ # Windows
+ >tools/export-data.bat -t ./data/ -q 'select * from root.stock.**'
+ ```
+
+- 导出结果
+ ```Bash
+
Time,root.stock.Legacy.0700HK.L1_BidPrice,root.stock.Legacy.0700HK.Type,root.stock.Legacy.0700HK.L1_BidSize,root.stock.Legacy.0700HK.Domain,root.stock.Legacy.0700HK.L1_BuyNo,root.stock.Legacy.0700HK.L1_AskPrice
+
2024-07-29T18:37:18.700+08:00,0.9666617,3.0,0.021367407654674264,-6.0,false,0.8926191
+
2024-07-29T18:37:19.701+08:00,0.3057328,3.0,0.9965377284981661,-5.0,false,0.15167356
+ ```
+- 导出 SQL 文件内所有 SQL 执行范围下的所有数据至 CSV 文件。
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -t ./data/ -s export.sql
+ # Windows
+ >tools/export-data.bat -t ./data/ -s export.sql
+ ```
+
+- export.sql 文件内容(-s 参数指向的文件)
+ ```SQL
+ select * from root.stock.** limit 100
+ select * from root.db.** limit 100
+ ```
+
+- 导出结果文件1
+ ```Bash
+
Time,root.stock.Legacy.0700HK.L1_BidPrice,root.stock.Legacy.0700HK.Type,root.stock.Legacy.0700HK.L1_BidSize,root.stock.Legacy.0700HK.Domain,root.stock.Legacy.0700HK.L1_BuyNo,root.stock.Legacy.0700HK.L1_AskPrice
+
2024-07-29T18:37:18.700+08:00,0.9666617,3.0,0.021367407654674264,-6.0,false,0.8926191
+
2024-07-29T18:37:19.701+08:00,0.3057328,3.0,0.9965377284981661,-5.0,false,0.15167356
+ ```
+
+- 导出结果文件2
+ ```Bash
+ Time,root.db.Random.RandomBoolean
+ 2024-07-22T17:16:05.820+08:00,true
+ 2024-07-22T17:16:02.597+08:00,false
+ ```
+- 将 IoTDB 数据库中在 SQL 文件内定义的数据,以对齐的格式将其导出为 SQL 语句。
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -h 127.0.0.1 -p 6667 -u root -p root -t ./data/ -s
export.sql -type sql -aligned true
+ # Windows
+ >tools/export-data.bat -h 127.0.0.1 -p 6667 -u root -p root -t ./data/ -s
export.sql -type sql -aligned true
+ ```
+
+- 导出结果
+ ```Bash
+ INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
ALIGNED VALUES
(1722249629831,0.62308747,2.0,0.012206747854849653,-6.0,false,0.14164352);
+ INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
ALIGNED VALUES
(1722249630834,0.7520042,3.0,0.22760657101910464,-5.0,true,0.089064896);
+ INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
ALIGNED VALUES
(1722249631835,0.3981064,3.0,0.6254559288663467,-6.0,false,0.9767922);
+ ```
+- 将某 SQL 执行范围下的所有数据导出至 CSV 文件,指定导出的时间格式为`yyyy-MM-dd
HH:mm:ss`,且表头时间序列的后面打印出对应的数据类型。
+ ```Bash
+ # Unix/OS X
+ >tools/export-data.sh -t ./data/ -tf 'yyyy-MM-dd HH:mm:ss' -datatype true
-q "select * from root.stock.**" -type csv
+ # Windows
+ >tools/export-data.bat -t ./data/ -tf 'yyyy-MM-dd HH:mm:ss' -datatype true
-q "select * from root.stock.**" -type csv
+ ```
+
+- 导出结果
+ ```Bash
+
Time,root.stock.Legacy.0700HK.L1_BidPrice(DOUBLE),root.stock.Legacy.0700HK.Type(DOUBLE),root.stock.Legacy.0700HK.L1_BidSize(DOUBLE),root.stock.Legacy.0700HK.Domain(DOUBLE),root.stock.Legacy.0700HK.L1_BuyNo(BOOLEAN),root.stock.Legacy.0700HK.L1_AskPrice(DOUBLE)
+ 2024-07-30 10:33:55,0.44574088,3.0,0.21476832811611501,-4.0,true,0.5951748
+ 2024-07-30 10:33:56,0.6880933,3.0,0.6289119476165305,-5.0,false,0.114634395
+ ```
+
+# import-data 脚本(数据导入)
+
+## 导入文件示例
+
+### CSV 文件示例
+
+注意,在导入 CSV 数据前,需要特殊处理下列的字符:
+
+1. 如果 Text 类型的字段中包含特殊字符如`,`需要使用`\`来进行转义。
+2. 可以导入像`yyyy-MM-dd'T'HH:mm:ss`, `yyy-MM-dd HH:mm:ss`, 或者
`yyyy-MM-dd'T'HH:mm:ss.SSSZ`格式的时间。
+3. 时间列`Time`应该始终放在第一列。
+
+示例一:时间对齐,并且 header 中不包含数据类型的数据。
+
+```SQL
+Time,root.test.t1.str,root.test.t2.str,root.test.t2.var
+1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
+1970-01-01T08:00:00.002+08:00,"123",,
+```
+
+示例二:时间对齐,并且 header 中包含数据类型的数据(Text 类型数据支持加双引号和不加双引号)
+
+```SQL
+Time,root.test.t1.str(TEXT),root.test.t2.str(TEXT),root.test.t2.var(INT32)
+1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
+1970-01-01T08:00:00.002+08:00,123,hello world,123
+1970-01-01T08:00:00.003+08:00,"123",,
+1970-01-01T08:00:00.004+08:00,123,,12
+```
+
+示例三:设备对齐,并且 header 中不包含数据类型的数据
+
+```SQL
+Time,Device,str,var
+1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
+1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
+1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
+```
+
+示例四:设备对齐,并且 header 中包含数据类型的数据(Text 类型数据支持加双引号和不加双引号)
+
+```SQL
+Time,Device,str(TEXT),var(INT32)
+1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
+1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
+1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
+1970-01-01T08:00:00.002+08:00,root.test.t1,hello world,123
+```
+
+### SQL 文件示例
+
+> 对于不支持的 SQL ,不合法的 SQL,执行失败的 SQL 都会放到失败目录下的失败文件里(默认为 文件名.failed)
+
+```SQL
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728578812,0.21911979,4.0,0.7129878488375604,-5.0,false,0.65362453);
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728579812,0.35814416,3.0,0.04674720094979623,-5.0,false,0.9365247);
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728580813,0.20012152,3.0,0.9910098187911393,-4.0,true,0.70040536);
+INSERT INTO
root.stock.Legacy.0700HK(TIMESTAMP,L1_BidPrice,Type,L1_BidSize,Domain,L1_BuyNo,L1_AskPrice)
VALUES (1721728581814,0.034122765,4.0,0.9313345284181858,-4.0,true,0.9945297);
+```
+
+## 运行命令
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -h <ip> -p <port> -u <username> -pw <password> -s
<xxx.csv/sql> [-fd <./failedDirectory> -aligned <true/false> -batch <int> -tp
<ms/ns/us> -typeInfer <boolean=text,float=double...> -lpf <int>]
+
+# Windows
+>tools\import-data.bat -h <ip> -p <port> -u <username> -pw <password> -s
<xxx.csv/sql> [-fd <./failedDirectory> -aligned <true/false> -batch <int> -tp
<ms/ns/us> -typeInfer <boolean=text,float=double...> -lpf <int>]
+```
+
+> 虽然 IoTDB 具有类型推断的能力,但我们仍然推荐在导入数据前创建元数据,因为这可以避免不必要的类型转换错误。如下:
+
+```SQL
+CREATE DATABASE root.fit.d1;
+CREATE DATABASE root.fit.d2;
+CREATE DATABASE root.fit.p;
+CREATE TIMESERIES root.fit.d1.s1 WITH DATATYPE=INT32,ENCODING=RLE;
+CREATE TIMESERIES root.fit.d1.s2 WITH DATATYPE=TEXT,ENCODING=PLAIN;
+CREATE TIMESERIES root.fit.d2.s1 WITH DATATYPE=INT32,ENCODING=RLE;
+CREATE TIMESERIES root.fit.d2.s3 WITH DATATYPE=INT32,ENCODING=RLE;
+CREATE TIMESERIES root.fit.p.s1 WITH DATATYPE=INT32,ENCODING=RLE;
+```
+
+参数介绍:
+
+| 参数 | 定义
[...]
+|:---------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| -h | 数据库IP地址
[...]
+| -p | 数据库端口
[...]
+| -u | 数据库连接用户名
[...]
+| -pw | 数据库连接密码
[...]
+| -s |
指定想要导入的数据,这里可以指定文件或者文件夹。如果指定的是文件夹,将会把文件夹中所有的后缀为csv或者sql的文件进行批量导入。
[...]
+| -fd |
指定存放失败sql文件的目录,如果未指定这个参数,失败的文件将会被保存到源数据的目录中。<br/>说明:对于不支持的 SQL,不合法的 SQL,执行失败的
SQL 都会放到失败目录下的失败文件里(默认为 文件名.failed)
[...]
+| -aligned | 指定是否使用`aligned`接口,选项为 true 或者 false
<br/>说明:这个参数只在导入文件为 csv 文件时生效
[...]
+| -batch |
用于指定每一批插入的数据的点数(最小值为1,最大值为Integer.*MAX_VALUE*)。如果程序报了`org.apache.thrift.transport.TTransportException:
Frame size larger than protect max size`这个错的话,就可以适当的调低这个参数。
[...]
+| -tp | 指定时间精度,可选值包括`ms`(毫秒),`ns`(纳秒),`us`(微秒)
[...]
+| -lpf | 指定每个导入失败文件写入数据的行数
[...]
+| -typeInfer |
用于指定类型推断规则,如<srcTsDataType1=dstTsDataType1,srcTsDataType2=dstTsDataType2,...>。<br/>说明:用于指定类型推断规则.`srcTsDataType`
包括 `boolean`,`int`,`long`,`float`,`double`,`NaN`.`dstTsDataType` 包括
`boolean`,`int`,`long`,`float`,`double`,`text`.当`srcTsDataType`为`boolean`,
`dstTsDataType`只能为`boolean`或`text`.当`srcTsDataType`为`NaN`,
`dstTsDataType`只能为`float`, `double`或`text`.当`srcTsDataType`为数值类型,
`dstTsDataType`的精度需要高于`srcTsDataType`.例如:`-typeInfer boolean=text,float=double`
[...]
+
+## 运行示例
+
+- 导入当前`data`目录下的`dump0_0.sql`数据到本机 IoTDB 数据库中。
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -s ./data/dump0_0.sql
+# Windows
+>tools/import-data.bat -s ./data/dump0_0.sql
+```
+
+- 将当前`data`目录下的所有数据以对齐的方式导入到本机 IoTDB 数据库中。
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -s ./data/ -fd ./failed/ -aligned true
+# Windows
+>tools/import-data.bat -s ./data/ -fd ./failed/ -aligned true
+```
+
+- 导入当前`data`目录下的`dump0_0.csv`数据到本机 IoTDB 数据库中。
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -s ./data/dump0_0.csv -fd ./failed/
+# Windows
+>tools/import-data.bat -s ./data/dump0_0.csv -fd ./failed/
+```
+
+- 将当前`data`目录下的`dump0_0.csv`数据以对齐的方式,一批导入100000条导入到`192.168.100.1`IP所在主机的
IoTDB 数据库中,失败的记录记在当前`failed`目录下,每个文件最多记1000条。
+
+```Bash
+# Unix/OS X
+>tools/import-data.sh -h 192.168.100.1 -p 6667 -u root -pw root -s
./data/dump0_0.csv -fd ./failed/ -aligned true -batch 100000 -tp ms -typeInfer
boolean=text,float=double -lpf 1000
+# Windows
+>tools/import-data.bat -h 192.168.100.1 -p 6667 -u root -pw root -s
./data/dump0_0.csv -fd ./failed/ -aligned true -batch 100000 -tp ms -typeInfer
boolean=text,float=double -lpf 1000
+```
\ No newline at end of file
diff --git a/src/zh/UserGuide/latest/Tools-System/Import-Export-Tool.md
b/src/zh/UserGuide/latest/Tools-System/TsFile-Import-Export-Tool.md
similarity index 99%
rename from src/zh/UserGuide/latest/Tools-System/Import-Export-Tool.md
rename to src/zh/UserGuide/latest/Tools-System/TsFile-Import-Export-Tool.md
index 28b0b98..3e83baf 100644
--- a/src/zh/UserGuide/latest/Tools-System/Import-Export-Tool.md
+++ b/src/zh/UserGuide/latest/Tools-System/TsFile-Import-Export-Tool.md
@@ -19,17 +19,17 @@
-->
-# 导入导出工具
+# TsFile 导入导出脚本
针对于不同场景,IoTDB 为用户提供多种批量导入数据的操作方式,本章节向大家介绍最为常用的两种方式为 CSV文本形式的导入 和 TsFile文件形式的导入。
-## TsFile导入导出工具
+## TsFile 导入导出脚本
TsFile 是在 IoTDB 中使用的时间序列的文件格式,您可以通过CLI等工具直接将存有时间序列的一个或多个 TsFile
文件导入到另外一个正在运行的IoTDB实例中。
### 介绍
加载外部 tsfile 文件工具允许用户向正在运行中的 Apache IoTDB 中加载 tsfile
文件。或者您也可以使用脚本的方式将tsfile加载进IoTDB。
-### 使用SQL加载
+### 使用 SQL 加载
用户通过 Cli 工具或 JDBC 向 Apache IoTDB 系统发送指定命令实现文件加载的功能。
#### 加载 tsfile 文件
diff --git a/src/zh/UserGuide/latest/User-Manual/Write-Delete-Data.md
b/src/zh/UserGuide/latest/User-Manual/Write-Delete-Data.md
index d3b7ada..0966d28 100644
--- a/src/zh/UserGuide/latest/User-Manual/Write-Delete-Data.md
+++ b/src/zh/UserGuide/latest/User-Manual/Write-Delete-Data.md
@@ -176,11 +176,11 @@ It costs 0.004s
### TsFile批量导入
-TsFile 是在 IoTDB 中使用的时间序列的文件格式,您可以通过CLI等工具直接将存有时间序列的一个或多个 TsFile
文件导入到另外一个正在运行的IoTDB实例中。具体操作方式请参考[导入导出工具](../Tools-System/Import-Export-Tool.md)。
+TsFile 是在 IoTDB 中使用的时间序列的文件格式,您可以通过CLI等工具直接将存有时间序列的一个或多个 TsFile
文件导入到另外一个正在运行的IoTDB实例中。具体操作方式请参考[导入导出工具](../Tools-System/TsFile-Import-Export-Tool.md)。
### CSV批量导入
-CSV 是以纯文本形式存储表格数据,您可以在CSV文件中写入多条格式化的数据,并批量的将这些数据导入到 IoTDB
中,在导入数据之前,建议在IoTDB中创建好对应的元数据信息。如果忘记创建元数据也不要担心,IoTDB
可以自动将CSV中数据推断为其对应的数据类型,前提是你每一列的数据类型必须唯一。除单个文件外,此工具还支持以文件夹的形式导入多个 CSV
文件,并且支持设置如时间精度等优化参数。具体操作方式请参考[导入导出工具](../Tools-System/Import-Export-Tool.md)。
+CSV 是以纯文本形式存储表格数据,您可以在CSV文件中写入多条格式化的数据,并批量的将这些数据导入到 IoTDB
中,在导入数据之前,建议在IoTDB中创建好对应的元数据信息。如果忘记创建元数据也不要担心,IoTDB
可以自动将CSV中数据推断为其对应的数据类型,前提是你每一列的数据类型必须唯一。除单个文件外,此工具还支持以文件夹的形式导入多个 CSV
文件,并且支持设置如时间精度等优化参数。具体操作方式请参考[导入导出工具](../Tools-System/Data-Import-Export-Tool.md)。
## 删除数据