Hi all,
I'd like to share a new update we are introducing to improve our data
export capabilities. Often, users need a straightforward and efficient way
to dump their relational query results into a standard format for backups,
migrations, or offline analysis.
To address this, this pull request adds support for using the `COPY TO`
statement to export query results directly into the TsFile format when
operating in Table Mode. Under the hood, we've implemented the necessary
execution operators, including `TableCopyToOperator` and
`TsFileFormatCopyToWriter`, to handle the formatting and writing processes.
We've also updated the query planner to properly generate these operators
and added integration tests to ensure the exports work reliably.
COPY TO syntax is from PG[1] and DuckDB[2].
Our complete syntax is:
```
copyToStatement
: COPY '(' query ')' TO fileName=string ((WITH)?
copyToStatementOptions)?
| COPY tableName=qualifiedName ('(' tableColumns=identifierList ')')?
TO fileName=string ((WITH)? copyToStatementOptions)?
;
copyToStatementOptions
: '(' copyToStatementOption (',' copyToStatementOption)* ')'
;
copyToStatementOption
: FORMAT identifier
| TABLE identifier
| TAGS '(' identifierList ')'
| TIME identifier
| MEMORY_THRESHOLD memory=INTEGER_VALUE
;
```
You can review the changes here: https://github.com/apache/iotdb/pull/17372
[1] https://duckdb.org/docs/stable/data/parquet/overview
[2] https://www.postgresql.org/docs/18/sql-copy.html
Best regards,
--------------------
Yuan Tian
