CritasWang commented on code in PR #18828:
URL: https://github.com/apache/iotdb/pull/18828#discussion_r4332623740


##########
iotdb-client/cli/src/main/java/org/apache/iotdb/tool/data/AbstractImportData.java:
##########
@@ -34,10 +37,39 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Objects;
+import java.util.concurrent.atomic.AtomicBoolean;
 
 public abstract class AbstractImportData extends AbstractDataTool implements 
Runnable {
 
   private static final IoTPrinter ioTPrinter = new IoTPrinter(System.out);
+  private static final AtomicBoolean importFailed = new AtomicBoolean();

Review Comment:
   `importFailed` is private and is only set inside `AbstractImportData`: on 
worker exceptions, archival failures and failed TsFile loads. These paths still 
print "Import completely!" and exit with 0:
   - **`.sql` input.** Failed statements in `ImportDataTree#importFromSqlFile` 
/ `ImportDataTable#importFromSqlFile` are only written to the `.failed` file. 
In a probe where every statement throws `StatementExecutionException`, `init()` 
still returns `true`.
   - **CSV files rejected as a whole.** This covers an empty file, a first 
column that is not `Time`, a wrong extension, and an `IOException` while 
reading, i.e. the early returns and the `catch` in both `importFromCsvFile` 
implementations. A probe with the header `ts,root.a.d1.s1` returns `true`.
   
   Could we add something like `protected static void markImportFailed()` and 
call it from those branches? If that is out of scope on purpose, it would help 
to state in the description which failure modes now produce a non-zero exit 
code.
   
   Whether row-level conversion failures that only go to `.failed` should also 
fail the run is a separate product decision, so I'm not suggesting that here.



##########
iotdb-client/cli/src/main/java/org/apache/iotdb/tool/data/AbstractImportData.java:
##########
@@ -65,15 +97,20 @@ public void run() {
         }
       }
     } catch (Exception e) {
+      importFailed.set(true);
+      loadFileFailedNum.increment();
       ioTPrinter.println(
-          String.format("[%s] - Unexpected error occurred: %s", 
file.getName(), e.getMessage()));
+          String.format(
+              CliMessages.MESSAGE_ARG_UNEXPECTED_ERROR_OCCURRED_ARG_E7255EE6,
+              file == null ? filePath : file.getName(),
+              e.getMessage()));
     }

Review Comment:
   This `catch` wraps the whole `while (... pollFromQueue() != null)` loop. 
`executeWithRetry` now throws `IllegalStateException`, but `importFromCsvFile` 
only catches `IOException` (plus `IllegalPathException` in the tree variant). 
As a result, a single failed batch ends this worker thread.
   
   The files still in the queue are then not imported, not counted in 
`loadFileFailedNum`, and not mentioned in the output:
   - With `-tn 1`, or once every thread has hit a failure, all remaining files 
are skipped.
   - With the default 8 threads, which files get imported depends on how many 
threads have already exited.
   
   For the tree align-by-device path this is also a behaviour change, because 
it used to log the `StatementExecutionException` and continue.
   
   Repro: set `threadNum = 1` and queue `a_bad.csv` (its batch throws), then 
`b_good.csv`. `init()` returns `false`, but `b_good.csv` is still in 
`ImportDataScanTool`'s queue afterwards and was never inserted.
   
   Suggestion: move the `try`/`catch` inside the loop body, for both the TsFile 
loop and the CSV/SQL loop. Each failure is then handled per file: mark the 
import as failed, increment `loadFileFailedNum`, print the file name, and 
continue with the next file.
   
   <details><summary>Probe test (fits into <code>DataToolRegressionTest</code>; 
fails today)</summary>
   
   ```java
   @Test
   public void testFailedCsvDoesNotSkipRemainingFiles() throws Exception {
     Path bad = directory.resolve("a_bad.csv");
     Path good = directory.resolve("b_good.csv");
     Files.writeString(bad, "Time,root.a.d1.s1(INT32)\n1,11\n");
     Files.writeString(good, "Time,root.a.d2.s1(INT32)\n1,22\n");
     SessionPool pool = mock(SessionPool.class);
     set(ImportDataTree.class, "sessionPool", pool);
     doAnswer(
             invocation -> {
               if (((List<?>) invocation.getArgument(0)).contains("root.a.d1")) 
{
                 throw new StatementExecutionException("type mismatch");
               }
               return null;
             })
         .when(pool)
         .insertRecords(any(), any(), any(), any(), any());
     ImportDataScanTool.putToQueue(bad.toString());
     ImportDataScanTool.putToQueue(good.toString());
     assertFalse(AbstractImportData.init(new ImportDataTree())); // threadNum = 
1 in setUp()
     assertNull(ImportDataScanTool.pollFromQueue()); // fails today: b_good.csv 
is still queued
   }
   ```
   
   </details>



##########
iotdb-client/cli/src/main/java/org/apache/iotdb/tool/data/ExportDataTable.java:
##########
@@ -131,31 +147,108 @@ public void exportBySql(String sql, int index) {
       }
     }
     for (int i = 0; i < exportSql.size(); i++) {
-      String path = targetDirectory + targetFile + i;
-      String table = tables.get(i);
-      try (SessionDataSet sessionDataSet =
-          tableSession.executeQueryStatement(exportSql.get(i), timeout)) {
-        if (Constants.SQL_SUFFIXS.equalsIgnoreCase(exportType)) {
-          exportToSqlFile(sessionDataSet, table, path);
-        } else if (Constants.TSFILE_SUFFIXS.equalsIgnoreCase(exportType)) {
-          long start = System.currentTimeMillis();
-          boolean isComplete = exportToTsFile(sessionDataSet, path + 
".tsfile", table);
-          if (isComplete) {
-            long end = System.currentTimeMillis();
-            ioTPrinter.println("Export completely!cost: " + (end - start) + " 
ms.");
+      String path = targetDirectory + targetFile + (customQuery ? index : i);
+      try {
+        String queryTable =
+            customQuery
+                ? (Constants.SQL_SUFFIXS.equalsIgnoreCase(exportType)
+                        || 
Constants.TSFILE_SUFFIXS.equalsIgnoreCase(exportType)
+                    ? getQueryTable(sql)
+                    : null)
+                : tables.get(i);
+        try (SessionDataSet sessionDataSet =
+            tableSession.executeQueryStatement(exportSql.get(i), timeout)) {
+          if (Constants.SQL_SUFFIXS.equalsIgnoreCase(exportType)) {
+            exportToSqlFile(sessionDataSet, queryTable, path);
+          } else if (Constants.TSFILE_SUFFIXS.equalsIgnoreCase(exportType)) {
+            long start = System.currentTimeMillis();
+            boolean isComplete = exportToTsFile(sessionDataSet, path + 
".tsfile", queryTable);
+            if (isComplete) {
+              long end = System.currentTimeMillis();
+              ioTPrinter.println("Export completely!cost: " + (end - start) + 
" ms.");
+            }
+          } else {
+            exportToCsvFile(sessionDataSet, path);
           }
-        } else {
-          exportToCsvFile(sessionDataSet, path);
+          sessionDataSet.closeOperationHandle();
+          ioTPrinter.println(Constants.EXPORT_COMPLETELY);
         }
-        sessionDataSet.closeOperationHandle();
-        ioTPrinter.println(Constants.EXPORT_COMPLETELY);
       } catch (StatementExecutionException
           | IoTDBConnectionException
           | IOException
           | WriteProcessException e) {
         ioTPrinter.println("Cannot dump result because: " + e.getMessage());
+        success = false;
       }
     }
+    return success;
+  }
+
+  private static String getQueryTable(String sql) throws 
StatementExecutionException {
+    String message =
+        String.format(
+            CliMessages
+                
.EXCEPTION_SQL_TSFILE_EXPORT_REQUIRES_A_QUERY_FROM_A_SINGLE_TABLE_IN_DATABASE_ARG_C9EB3768,
+            database);
+    RelationalSqlLexer lexer =
+        new RelationalSqlLexer(new 
CaseInsensitiveStream(CharStreams.fromString(sql)));
+    RelationalSqlParser parser = new RelationalSqlParser(new 
CommonTokenStream(lexer));
+    lexer.removeErrorListeners();
+    lexer.addErrorListener(
+        new BaseErrorListener() {
+          @Override
+          public void syntaxError(
+              Recognizer<?, ?> recognizer,
+              Object offendingSymbol,
+              int line,
+              int charPositionInLine,
+              String error,
+              RecognitionException cause) {
+            throw new ParseCancellationException(message);
+          }
+        });
+    parser.removeErrorListeners();
+    parser.setErrorHandler(new BailErrorStrategy());
+    List<RelationalSqlParser.QualifiedNameContext> sources = new ArrayList<>();
+    try {
+      new RelationalSqlBaseVisitor<Void>() {
+        @Override
+        public Void visitTableName(RelationalSqlParser.TableNameContext 
context) {
+          sources.add(context.qualifiedName());
+          return null;
+        }
+      }.visit(parser.singleStatement());
+    } catch (ParseCancellationException e) {
+      throw new StatementExecutionException(message);
+    }
+    // Multiple source relations do not have an unambiguous destination table.
+    if (sources.size() != 1) {

Review Comment:
   Minor / FYI: because this counts `tableName` nodes, it also rejects 
single-table queries that mention the table more than once, for example `select 
* from t where v in (select v from t)` or a CTE over `t`. Both fail with 
"requires a query from a single table". Failing closed seems right here, since 
simple de-duplication would also let self-joins through. It may be worth 
spelling out the limitation in the docs or help text, e.g. that joins, 
subqueries and CTEs are not supported for `-ft sql/tsfile` custom queries.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to