Hi all, I'd like to share an improvement introduced by the following PR: - master: https://github.com/apache/iotdb/pull/18928 COPY TO TsFile used to export the inner query result as-is, so the export failed whenever a query returned timestamps out of order within one target device, for example ORDER BY time DESC or an ordering on a field. Unlike a normal query, an export also had no way to be inspected or profiled: EXPLAIN / EXPLAIN ANALYZE did not accept COPY ... TO .... This PR makes COPY TO tolerant to unordered results and adds EXPLAIN and EXPLAIN ANALYZE support for both the table shorthand and SQL query exports. The previous behavior came from two limitations: - the TsFile writer requires non-decreasing timestamps per device, but the inner query result was piped to it directly, so any out-of-order result (descending time, field-based ordering, distributed merge) was rejected; - the parser, analyzer and planner only accepted a plain query or EXECUTE under EXPLAIN / EXPLAIN ANALYZE, so a COPY TO statement could not be explained or analyzed. The fix works as follows: - after planning the complete inner query, an outer sort on the target TAG columns and TIME is added, so the writer always receives rows in a valid order; the new RemoveRedundantSortForCopyTo drops a superseded inner sort, and the existing ordering optimizations reuse streaming scans, eliminate redundant sorts and merge ordered region results; - inner ORDER BY / LIMIT / OFFSET semantics are preserved; exports with generated timestamps keep their behavior, duplicate timestamps are still validated by the writer, and no new SQL option is added; - EXPLAIN / EXPLAIN ANALYZE now accept COPY TO: EXPLAIN prints the plan without creating a file, EXPLAIN ANALYZE runs the export, creates the TsFile and returns runtime statistics; text/JSON output and VERBOSE are supported where applicable; - IOException during target-file creation / writer initialization is normalized to COPY_TO_WRITE_ERROR (725), and a specified TIME column missing from the query result is rejected with a clear error. Main changes include: - added the RemoveRedundantSortForCopyTo optimizer and registered it before scan ordering and limit pushdown; - extended TableLogicalPlanner to build the outer TAG/TIME sort and the CopyToNode output symbols, and generalized the EXPLAIN / EXPLAIN ANALYZE planning; - allowed copyToStatement in the EXPLAIN / EXPLAIN ANALYZE grammar and parsed it in AstBuilder, and let StatementAnalyzer process COPY under EXPLAIN; - added output symbols to CopyToNode, and printed the CopyTo properties (target file, format, target table, TIME/TAG columns, memory threshold) in the plan text and JSON printers; - normalized COPY TO write errors and added the target TIME column existence check with matching i18n messages; - added planner unit tests and integration tests: descending/field ordering, LIMIT/OFFSET, aliased TIME with reordered TAGs, exports without TAGs, and EXPLAIN / EXPLAIN ANALYZE in text/JSON/VERBOSE, using small TsBlocks to exercise multiple batches. Feedback and suggestions are welcome.
大家好， 我想分享以下 PR 中做出的一项改进： - master：https://github.com/apache/iotdb/pull/18928 COPY TO TsFile 以往会直接把内层查询结果导出，因此只要查询在同一目标 device 内返回的 timestamp 不是递增的（例如 ORDER BY time DESC，或按字段排序），导出就会失败。同时导出也无法像普通查询那样被查看或分析：EXPLAIN / EXPLAIN ANALYZE 不接受 COPY ... TO ...。这个 PR 让 COPY TO 能够处理乱序结果，并为表简写和 SQL 查询两种导出方式都加上了 EXPLAIN 与 EXPLAIN ANALYZE 支持。 此前的问题源自两个限制： - TsFile writer 要求同一 device 内 timestamp 非递减，而内层查询结果被直接喂给了 writer，所以任何乱序结果（时间倒序、按字段排序、分布式归并）都会被拒绝； - 解析、分析和规划阶段在 EXPLAIN / EXPLAIN ANALYZE 下只接受普通查询或 EXECUTE，因此 COPY TO 语句无法被 explain 或 analyze。 本次改进的做法是： - 在完整规划完内层查询之后，追加一个按目标 TAG 列和 TIME 的外层排序，保证 writer 总是按合法顺序收到数据；新增的 RemoveRedundantSortForCopyTo 会去掉被取代的内层排序，已有的排序优化则复用流式扫描、消除冗余排序并归并有序的分区结果； - 保留内层 ORDER BY / LIMIT / OFFSET 的语义；生成时间戳的导出保持原有行为，重复时间戳仍由 writer 校验，且不新增任何 SQL 选项； - EXPLAIN / EXPLAIN ANALYZE 现在支持 COPY TO：EXPLAIN 只打印计划、不创建文件，EXPLAIN ANALYZE 会真正执行导出、生成 TsFile 并返回运行时统计；在适用的场景下支持 text/JSON 输出和 VERBOSE； - 目标文件创建 / writer 初始化阶段的 IOException 统一转换为 COPY_TO_WRITE_ERROR (725)；指定的 TIME 列在查询结果中不存在时会给出明确报错。 主要修改包括： - 新增 RemoveRedundantSortForCopyTo 优化器，并注册在扫描排序与 limit 下推之前； - 扩展 TableLogicalPlanner，构造外层 TAG/TIME 排序和 CopyToNode 的输出符号，并统一 EXPLAIN / EXPLAIN ANALYZE 的规划逻辑； - 在 EXPLAIN / EXPLAIN ANALYZE 语法中允许 copyToStatement，并在 AstBuilder 中解析，StatementAnalyzer 支持被 EXPLAIN 包裹的 COPY； - CopyToNode 增加输出符号，计划文本与 JSON 打印器输出 CopyTo 属性（目标文件、格式、目标表、TIME/TAG 列、内存阈值）； - 统一 COPY TO 的写错误，并新增目标 TIME 列存在性校验及对应 i18n 消息； - 新增规划器单元测试和集成测试：时间倒序/按字段排序、LIMIT/OFFSET、TIME 别名与重排 TAG、无 TAG 导出，以及 text/JSON/VERBOSE 下的 EXPLAIN / EXPLAIN ANALYZE，并用较小的 TsBlock 覆盖多批次场景。 欢迎大家提出反馈和建议。 Best regards, Wenwei Shu
