CritasWang commented on code in PR #19150:
URL: https://github.com/apache/iotdb/pull/19150#discussion_r4548813968


##########
iotdb-client/cli/src/main/java/org/apache/iotdb/tool/schema/ImportSchemaTree.java:
##########
@@ -268,31 +365,56 @@ private static void writeScheme(
                     compressionTypeRaw));
             failedRecords.add(recordObj.stream().collect(Collectors.toList()));
             failed = true;
+          } else if (!tagsParsed) {
+            failedRecords.add(recordObj.stream().collect(Collectors.toList()));
+            failed = true;
           } else {
             if (StringUtils.isBlank(alias)) {
               paths.add(path);
               dataTypes.add(dataType);
               encodings.add(encodingType);
               compressors.add(compressionType);
+              if (v2) {
+                tags.add(recordTags);
+                attributes.add(recordAttributes);
+              }
             } else {
               pathsWithAlias.add(path);
               dataTypesWithAlias.add(dataType);
               encodingsWithAlias.add(encodingType);
               compressorsWithAlias.add(compressionType);
               measurementAlias.add(alias);
+              if (v2) {
+                tagsWithAlias.add(recordTags);
+                attributesWithAlias.add(recordAttributes);
+              }
             }
             pointSize.getAndIncrement();
           }
           if (!failed && aligned) {
             String deviceId = path.substring(0, path.lastIndexOf("."));
             paths.add(0, path.substring(deviceId.length() + 1));
+            if (v2) {
+              // keep the maps aligned with the measurements list, whose head 
now holds this one
+              tags.add(0, recordTags);

Review Comment:
   Fixed in 49d840953a.
   
   `import-schema` now registers `-aligned` / `--use_aligned`. Each valid 
aligned row sends exactly one measurement and one corresponding entry for type, 
encoding, compression, alias, tags, and attributes. `MeasurementPath` preserves 
quoted measurement names containing dots. The existing incorrect mock 
expectation has been corrected.
   
   Added `alignedImportThroughCliPreservesEveryRow`, which runs the actual 
import script against the server for both v2 and legacy CSV files. It covers 
aliases, missing metadata, multiple measurements/devices, and a quoted 
measurement name, compares every schema field read back from the server, and 
checks that the devices are aligned.
   
   Passed with both `SimpleIT` (1C1D) and `ClusterIT` (1C3D).
   



##########
iotdb-client/cli/src/main/java/org/apache/iotdb/tool/schema/AbstractSchemaTool.java:
##########
@@ -218,32 +228,71 @@ public static Boolean writeCsvFile(List<List<Object>> 
records, String filePath)
     }
   }
 
+  /**
+   * The RFC4180 CSV layout of the schema files, i.e. the delimiter is {@code 
','}, the quote char
+   * is {@code '"'} and no escape char is configured. A JSON object cell is 
therefore always quoted
+   * and its inner quotes are doubled, which is exactly what RFC4180 asks for.
+   */
+  static CSVFormat.Builder createRfc4180CsvFormat() {
+    return CSVFormat.Builder.create(CSVFormat.DEFAULT)
+        .setQuote('"')
+        .setQuoteMode(QuoteMode.MINIMAL)
+        .setRecordSeparator('\n');
+  }
+
+  /**
+   * Serializes tags or attributes into the compact JSON object cell of the 
schema file. Keys are
+   * sorted so that repeated exports of the same schema are identical, and 
HTML escaping is disabled
+   * so that printable characters such as {@code '}, {@code =}, {@code <} and 
{@code &} stay
+   * readable.
+   *
+   * @return an empty string when there is nothing to write, never {@code {}} 
or {@code null}
+   */
+  static String toJsonCell(Map<String, String> values) {
+    if (values == null || values.isEmpty()) {
+      return "";
+    }
+    return JSON_GSON.toJson(new TreeMap<>(values));
+  }
+
+  /**
+   * Parses a Tags / Attributes cell written by {@link #toJsonCell(Map)}.
+   *
+   * @return {@code null} when the cell carries no value, never an empty map
+   * @throws com.google.gson.JsonParseException when the cell is not a JSON 
object
+   */
+  static Map<String, String> parseJsonCell(String cell) {
+    if (cell == null) {
+      return null;
+    }
+    String trimmed = cell.trim();
+    if (trimmed.isEmpty() || "null".equals(trimmed)) {
+      return null;
+    }
+    Map<String, String> parsed = JSON_GSON.fromJson(trimmed, JSON_CELL_TYPE);

Review Comment:
   Fixed in 49d840953a.
   
   Metadata values are now decoded as `JsonElement` and checked to be strings 
before a row enters either import path. JSON nulls, numbers, booleans, arrays, 
and objects are rejected through the existing failed-records path. Empty 
metadata cells/objects and string values such as `"null"` remain supported, and 
duplicate-key rejection is preserved.
   
   Added unit coverage for batch isolation and 
`invalidMetadataIsIsolatedFromValidRowsThroughCli` for the real script/server 
path. The latter checks all five invalid value types in both Tags and 
Attributes, verifies the exact 10 rows written to the failed file, and confirms 
that valid rows, including an aliased row, are created correctly.
   
   Passed with both `SimpleIT` (1C1D) and `ClusterIT` (1C3D).
   



##########
iotdb-client/cli/src/main/java/org/apache/iotdb/tool/schema/ImportSchemaTree.java:
##########
@@ -131,18 +134,68 @@ protected void importSchemaFromCsvFile(File file) {
     }
   }
 
-  private static CSVParser readCsvFile(String path) throws IOException {
-    return CSVFormat.Builder.create(CSVFormat.DEFAULT)
-        .setHeader()
-        .setSkipHeaderRecord(true)
-        .setQuote('`')
-        .setEscape('\\')
-        .setIgnoreEmptyLines(true)
+  static CSVParser readCsvFile(String path) throws IOException {
+    CSVFormat.Builder builder =
+        CSVFormat.Builder.create(CSVFormat.DEFAULT)
+            .setHeader()
+            .setSkipHeaderRecord(true)
+            .setIgnoreEmptyLines(true);
+    if (isV2Format(readHeaderNames(path))) {
+      builder.setQuote('"');
+    } else {
+      builder.setQuote('`').setEscape('\\');
+    }
+    return builder
         .build()
-        .parse(new InputStreamReader(new FileInputStream(path)));
+        .parse(new InputStreamReader(new FileInputStream(path), 
StandardCharsets.UTF_8));
+  }
+
+  /**
+   * Reads the column names of a schema file without parsing it, so that the 
CSV settings of the
+   * file can be selected before it is read.
+   */
+  static List<String> readHeaderNames(String path) throws IOException {
+    try (BufferedReader reader =
+        new BufferedReader(
+            new InputStreamReader(new FileInputStream(path), 
StandardCharsets.UTF_8))) {
+      String headerLine = reader.readLine();

Review Comment:
   Fixed in 49d840953a.
   
   Format detection and CSV parsing now share the same UTF-8 reader, which 
strips an optional BOM at the start of the file. Detection skips leading empty 
lines consistently with the parser's `ignoreEmptyLines` setting. The separate 
RFC4180 and legacy backtick/backslash settings are retained.
   
   Regression coverage includes LF/CRLF prefixes, BOM with and without leading 
empty lines, quoted/unquoted v2 headers, Tags JSON containing a comma, and 
legacy escaping. The real CLI/server tests also import files with a BOM and 
leading empty lines; all three integration cases passed in both 1C1D and 1C3D 
environments.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to