HTHou commented on code in PR #19150:
URL: https://github.com/apache/iotdb/pull/19150#discussion_r4559694008


##########
iotdb-client/cli/src/main/java/org/apache/iotdb/tool/schema/ImportSchemaTree.java:
##########
@@ -210,15 +279,36 @@ private static void writeScheme(
               dataTypes.clear();
               encodings.clear();
               compressors.clear();
+              pathsWithAlias.clear();

Review Comment:
   **[P2] Record failures from the aliased batch before clearing it**
   
   With `-batch 2` and three aliased rows, an alias of `bad.alias` on the 
second row made the server reject the first two rows. The catch block reads 
`paths`, which is empty for an all-aliased batch. Only the third time series 
was created, but the CLI reported complete success and wrote no failed-record 
file.
   
   Please capture the failed rows from the aliased batch, including their alias 
and metadata, before clearing its buffers.



##########
iotdb-client/cli/src/main/java/org/apache/iotdb/tool/schema/ImportSchemaTree.java:
##########
@@ -393,6 +548,8 @@ private static void writeAndEmptyDataSetAligned(
             encodings,
             compressors,
             measurementAliasList,
+            tagsList,
+            attributesList,
             --retryTime);
       }

Review Comment:
   **[P1] Report aligned import failures when connection retries are exhausted**
   
   I ran `import-schema -aligned true` with a valid seven-column CSV and an 
unused local port. After exhausting its retries, the CLI printed `Import 
completely successful!`, exited with code `0`, and produced no failed-record 
file. The last `IoTDBConnectionException` falls through this branch and the row 
is cleared.
   
   Please propagate the final connection error to the command exit status and 
preserve the failed row for retry.



##########
iotdb-client/cli/src/main/java/org/apache/iotdb/tool/schema/AbstractSchemaTool.java:
##########
@@ -218,32 +230,84 @@ public static Boolean writeCsvFile(List<List<Object>> 
records, String filePath)
     }
   }
 
+  /**
+   * The RFC4180 CSV layout of the schema files, i.e. the delimiter is {@code 
','}, the quote char
+   * is {@code '"'} and no escape char is configured. A JSON object cell is 
therefore always quoted
+   * and its inner quotes are doubled, which is exactly what RFC4180 asks for.
+   */
+  static CSVFormat.Builder createRfc4180CsvFormat() {
+    return CSVFormat.Builder.create(CSVFormat.DEFAULT)
+        .setQuote('"')
+        .setQuoteMode(QuoteMode.MINIMAL)
+        .setRecordSeparator('\n');
+  }
+
+  /**
+   * Serializes tags or attributes into the compact JSON object cell of the 
schema file. Keys are
+   * sorted so that repeated exports of the same schema are identical, and 
HTML escaping is disabled
+   * so that printable characters such as {@code '}, {@code =}, {@code <} and 
{@code &} stay
+   * readable.
+   *
+   * @return an empty string when there is nothing to write, never {@code {}} 
or {@code null}
+   */
+  static String toJsonCell(Map<String, String> values) {
+    if (values == null || values.isEmpty()) {
+      return "";
+    }
+    return JSON_GSON.toJson(new TreeMap<>(values));
+  }
+
+  /**
+   * Parses a Tags / Attributes cell written by {@link #toJsonCell(Map)}.
+   *
+   * @return {@code null} when the cell carries no value, never an empty map
+   * @throws com.google.gson.JsonParseException when the cell is not a JSON 
object with string
+   *     values
+   */
+  static Map<String, String> parseJsonCell(String cell) {
+    if (cell == null) {
+      return null;
+    }
+    String trimmed = cell.trim();
+    if (trimmed.isEmpty() || "null".equals(trimmed)) {
+      return null;
+    }
+    Map<String, JsonElement> parsed = JSON_GSON.fromJson(trimmed, 
JSON_CELL_TYPE);

Review Comment:
   **[P3] Require a JSON object with strict JSON syntax**
   
   The Gson map adapter accepts `[["room","101"]]`, and the default parser 
accepts `{"room":yes}`. I verified that both cells import successfully, even 
though the format specifies a JSON object with string values.
   
   Please check the root token and use strict JSON parsing while preserving 
duplicate-key rejection and the supported empty-cell behavior.



##########
iotdb-client/cli/src/main/java/org/apache/iotdb/tool/schema/ImportSchemaTree.java:
##########
@@ -131,18 +134,78 @@ protected void importSchemaFromCsvFile(File file) {
     }
   }
 
-  private static CSVParser readCsvFile(String path) throws IOException {
-    return CSVFormat.Builder.create(CSVFormat.DEFAULT)
-        .setHeader()
-        .setSkipHeaderRecord(true)
-        .setQuote('`')
-        .setEscape('\\')
-        .setIgnoreEmptyLines(true)
-        .build()
-        .parse(new InputStreamReader(new FileInputStream(path)));
+  static CSVParser readCsvFile(String path) throws IOException {
+    CSVFormat.Builder builder =
+        CSVFormat.Builder.create(CSVFormat.DEFAULT)
+            .setHeader()
+            .setSkipHeaderRecord(true)
+            .setIgnoreEmptyLines(true);
+    if (isV2Format(readHeaderNames(path))) {
+      builder.setQuote('"');
+    } else {
+      builder.setQuote('`').setEscape('\\');
+    }
+    return builder.build().parse(openCsvReader(path));
+  }
+
+  private static BufferedReader openCsvReader(String path) throws IOException {
+    BufferedReader reader =
+        new BufferedReader(
+            new InputStreamReader(new FileInputStream(path), 
StandardCharsets.UTF_8));
+    reader.mark(1);
+    if (reader.read() != '\uFEFF') {
+      reader.reset();
+    }
+    return reader;
+  }
+
+  /**
+   * Reads the column names of a schema file without parsing it, so that the 
CSV settings of the
+   * file can be selected before it is read.
+   */
+  static List<String> readHeaderNames(String path) throws IOException {
+    try (BufferedReader reader = openCsvReader(path)) {
+      String headerLine;
+      do {
+        headerLine = reader.readLine();
+      } while (headerLine != null && headerLine.isEmpty());
+      if (headerLine == null) {
+        return Collections.emptyList();
+      }
+      return Arrays.stream(headerLine.split(","))
+          .map(ImportSchemaTree::unquoteColumnName)
+          .collect(Collectors.toList());
+    }
   }
 
-  private static boolean checkHeader(List<String> headerNames) {
+  /** Accepts a header written by a spreadsheet tool, which quotes every 
column name. */
+  private static String unquoteColumnName(String columnName) {
+    String name = columnName.trim();
+    if (name.length() > 1 && name.startsWith("\"") && name.endsWith("\"")) {
+      return name.substring(1, name.length() - 1).trim();
+    }
+    return name;
+  }
+
+  /**
+   * A file whose header carries a Tags or an Attributes column is a v2 file, 
which is RFC4180 and
+   * holds JSON cells. Everything else is a legacy file and keeps the settings 
it was written with.
+   */
+  static boolean isV2Format(List<String> headerNames) {
+    return headerNames.contains(Constants.HEADER_TAGS)
+        || headerNames.contains(Constants.HEADER_ATTRIBUTES);
+  }
+
+  static boolean checkHeader(List<String> headerNames) {
+    if (isV2Format(headerNames)) {

Review Comment:
   **[P2] Keep format detection and header validation consistent**
   
   A seven-column CSV with headers named ` Tags` and ` Attributes` is detected 
as V2 because the probe trims the names. The parser preserves the leading 
spaces, so validation and `writeScheme` treat the same file as legacy and 
silently discard both metadata columns. I verified that the CLI reported 
success and created the time series without its metadata.
   
   Please use the same normalized headers throughout, or explicitly reject the 
file when a V2 header fails validation.



##########
iotdb-client/cli/src/main/java/org/apache/iotdb/tool/schema/ImportSchemaTree.java:
##########
@@ -267,32 +371,69 @@ private static void writeScheme(
                     path,
                     compressionTypeRaw));
             failedRecords.add(recordObj.stream().collect(Collectors.toList()));
-            failed = true;
+          } else if (!tagsParsed) {
+            failedRecords.add(recordObj.stream().collect(Collectors.toList()));
+          } else if (aligned) {
+            MeasurementPath timeseries;
+            try {
+              timeseries = new MeasurementPath(path);
+            } catch (IllegalPathException | IllegalArgumentException e) {
+              ioTPrinter.println(
+                  String.format(
+                      
CliMessages.MESSAGE_LINE_ARG_COLUMN_ARG_ILLEGAL_PATH_ARG_D5C1EBD6,
+                      recordObj.getRecordNumber(),
+                      headerNames,
+                      path));
+              
failedRecords.add(recordObj.stream().collect(Collectors.toList()));
+              return;
+            }
+            paths.add(timeseries.getMeasurement());
+            dataTypes.add(dataType);
+            encodings.add(encodingType);
+            compressors.add(compressionType);
+            measurementAlias.add(StringUtils.isBlank(alias) ? null : alias);
+            if (v2) {
+              tags.add(recordTags);

Review Comment:
   **[P3] Omit metadata parameters for aligned rows with empty metadata**
   
   For aligned rows with both cells empty, the lists still contain a null 
entry. `Session` converts the entries to empty maps for Thrift, so the server 
writes a tag log record for each row. Importing the same 12 aligned 
measurements left `tlog.txt` at 0 bytes with the five-column CSV and wrote 
8,400 bytes with the seven-column CSV.
   
   Please omit both metadata lists when the aligned row has neither tags nor 
attributes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to