CritasWang commented on code in PR #18828:
URL: https://github.com/apache/iotdb/pull/18828#discussion_r4332623740
##########
iotdb-client/cli/src/main/java/org/apache/iotdb/tool/data/AbstractImportData.java:
##########
@@ -34,10 +37,39 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
+import java.util.concurrent.atomic.AtomicBoolean;
public abstract class AbstractImportData extends AbstractDataTool implements
Runnable {
private static final IoTPrinter ioTPrinter = new IoTPrinter(System.out);
+ private static final AtomicBoolean importFailed = new AtomicBoolean();
Review Comment:
`importFailed` is private and is only set inside `AbstractImportData`: on
worker exceptions, archival failures and failed TsFile loads. These paths still
print "Import completely!" and exit with 0:
- **`.sql` input.** Failed statements in `ImportDataTree#importFromSqlFile`
/ `ImportDataTable#importFromSqlFile` are only written to the `.failed` file.
In a probe where every statement throws `StatementExecutionException`, `init()`
still returns `true`.
- **CSV files rejected as a whole.** This covers an empty file, a first
column that is not `Time`, a wrong extension, and an `IOException` while
reading, i.e. the early returns and the `catch` in both `importFromCsvFile`
implementations. A probe with the header `ts,root.a.d1.s1` returns `true`.
Could we add something like `protected static void markImportFailed()` and
call it from those branches? If that is out of scope on purpose, it would help
to state in the description which failure modes now produce a non-zero exit
code.
Whether row-level conversion failures that only go to `.failed` should also
fail the run is a separate product decision, so I'm not suggesting that here.
##########
iotdb-client/cli/src/main/java/org/apache/iotdb/tool/data/AbstractImportData.java:
##########
@@ -65,15 +97,20 @@ public void run() {
}
}
} catch (Exception e) {
+ importFailed.set(true);
+ loadFileFailedNum.increment();
ioTPrinter.println(
- String.format("[%s] - Unexpected error occurred: %s",
file.getName(), e.getMessage()));
+ String.format(
+ CliMessages.MESSAGE_ARG_UNEXPECTED_ERROR_OCCURRED_ARG_E7255EE6,
+ file == null ? filePath : file.getName(),
+ e.getMessage()));
}
Review Comment:
This `catch` wraps the whole `while (... pollFromQueue() != null)` loop.
`executeWithRetry` now throws `IllegalStateException`, but `importFromCsvFile`
only catches `IOException` (plus `IllegalPathException` in the tree variant).
As a result, a single failed batch ends this worker thread.
The files still in the queue are then not imported, not counted in
`loadFileFailedNum`, and not mentioned in the output:
- With `-tn 1`, or once every thread has hit a failure, all remaining files
are skipped.
- With the default 8 threads, which files get imported depends on how many
threads have already exited.
For the tree align-by-device path this is also a behaviour change, because
it used to log the `StatementExecutionException` and continue.
Repro: set `threadNum = 1` and queue `a_bad.csv` (its batch throws), then
`b_good.csv`. `init()` returns `false`, but `b_good.csv` is still in
`ImportDataScanTool`'s queue afterwards and was never inserted.
Suggestion: move the `try`/`catch` inside the loop body, for both the TsFile
loop and the CSV/SQL loop. Each failure is then handled per file: mark the
import as failed, increment `loadFileFailedNum`, print the file name, and
continue with the next file.
<details><summary>Probe test (fits into <code>DataToolRegressionTest</code>;
fails today)</summary>
```java
@Test
public void testFailedCsvDoesNotSkipRemainingFiles() throws Exception {
Path bad = directory.resolve("a_bad.csv");
Path good = directory.resolve("b_good.csv");
Files.writeString(bad, "Time,root.a.d1.s1(INT32)\n1,11\n");
Files.writeString(good, "Time,root.a.d2.s1(INT32)\n1,22\n");
SessionPool pool = mock(SessionPool.class);
set(ImportDataTree.class, "sessionPool", pool);
doAnswer(
invocation -> {
if (((List<?>) invocation.getArgument(0)).contains("root.a.d1"))
{
throw new StatementExecutionException("type mismatch");
}
return null;
})
.when(pool)
.insertRecords(any(), any(), any(), any(), any());
ImportDataScanTool.putToQueue(bad.toString());
ImportDataScanTool.putToQueue(good.toString());
assertFalse(AbstractImportData.init(new ImportDataTree())); // threadNum =
1 in setUp()
assertNull(ImportDataScanTool.pollFromQueue()); // fails today: b_good.csv
is still queued
}
```
</details>
##########
iotdb-client/cli/src/main/java/org/apache/iotdb/tool/data/ExportDataTable.java:
##########
@@ -131,31 +147,108 @@ public void exportBySql(String sql, int index) {
}
}
for (int i = 0; i < exportSql.size(); i++) {
- String path = targetDirectory + targetFile + i;
- String table = tables.get(i);
- try (SessionDataSet sessionDataSet =
- tableSession.executeQueryStatement(exportSql.get(i), timeout)) {
- if (Constants.SQL_SUFFIXS.equalsIgnoreCase(exportType)) {
- exportToSqlFile(sessionDataSet, table, path);
- } else if (Constants.TSFILE_SUFFIXS.equalsIgnoreCase(exportType)) {
- long start = System.currentTimeMillis();
- boolean isComplete = exportToTsFile(sessionDataSet, path +
".tsfile", table);
- if (isComplete) {
- long end = System.currentTimeMillis();
- ioTPrinter.println("Export completely!cost: " + (end - start) + "
ms.");
+ String path = targetDirectory + targetFile + (customQuery ? index : i);
+ try {
+ String queryTable =
+ customQuery
+ ? (Constants.SQL_SUFFIXS.equalsIgnoreCase(exportType)
+ ||
Constants.TSFILE_SUFFIXS.equalsIgnoreCase(exportType)
+ ? getQueryTable(sql)
+ : null)
+ : tables.get(i);
+ try (SessionDataSet sessionDataSet =
+ tableSession.executeQueryStatement(exportSql.get(i), timeout)) {
+ if (Constants.SQL_SUFFIXS.equalsIgnoreCase(exportType)) {
+ exportToSqlFile(sessionDataSet, queryTable, path);
+ } else if (Constants.TSFILE_SUFFIXS.equalsIgnoreCase(exportType)) {
+ long start = System.currentTimeMillis();
+ boolean isComplete = exportToTsFile(sessionDataSet, path +
".tsfile", queryTable);
+ if (isComplete) {
+ long end = System.currentTimeMillis();
+ ioTPrinter.println("Export completely!cost: " + (end - start) +
" ms.");
+ }
+ } else {
+ exportToCsvFile(sessionDataSet, path);
}
- } else {
- exportToCsvFile(sessionDataSet, path);
+ sessionDataSet.closeOperationHandle();
+ ioTPrinter.println(Constants.EXPORT_COMPLETELY);
}
- sessionDataSet.closeOperationHandle();
- ioTPrinter.println(Constants.EXPORT_COMPLETELY);
} catch (StatementExecutionException
| IoTDBConnectionException
| IOException
| WriteProcessException e) {
ioTPrinter.println("Cannot dump result because: " + e.getMessage());
+ success = false;
}
}
+ return success;
+ }
+
+ private static String getQueryTable(String sql) throws
StatementExecutionException {
+ String message =
+ String.format(
+ CliMessages
+
.EXCEPTION_SQL_TSFILE_EXPORT_REQUIRES_A_QUERY_FROM_A_SINGLE_TABLE_IN_DATABASE_ARG_C9EB3768,
+ database);
+ RelationalSqlLexer lexer =
+ new RelationalSqlLexer(new
CaseInsensitiveStream(CharStreams.fromString(sql)));
+ RelationalSqlParser parser = new RelationalSqlParser(new
CommonTokenStream(lexer));
+ lexer.removeErrorListeners();
+ lexer.addErrorListener(
+ new BaseErrorListener() {
+ @Override
+ public void syntaxError(
+ Recognizer<?, ?> recognizer,
+ Object offendingSymbol,
+ int line,
+ int charPositionInLine,
+ String error,
+ RecognitionException cause) {
+ throw new ParseCancellationException(message);
+ }
+ });
+ parser.removeErrorListeners();
+ parser.setErrorHandler(new BailErrorStrategy());
+ List<RelationalSqlParser.QualifiedNameContext> sources = new ArrayList<>();
+ try {
+ new RelationalSqlBaseVisitor<Void>() {
+ @Override
+ public Void visitTableName(RelationalSqlParser.TableNameContext
context) {
+ sources.add(context.qualifiedName());
+ return null;
+ }
+ }.visit(parser.singleStatement());
+ } catch (ParseCancellationException e) {
+ throw new StatementExecutionException(message);
+ }
+ // Multiple source relations do not have an unambiguous destination table.
+ if (sources.size() != 1) {
Review Comment:
Minor / FYI: because this counts `tableName` nodes, it also rejects
single-table queries that mention the table more than once, for example `select
* from t where v in (select v from t)` or a CTE over `t`. Both fail with
"requires a query from a single table". Failing closed seems right here, since
simple de-duplication would also let self-joins through. It may be worth
spelling out the limitation in the docs or help text, e.g. that joins,
subqueries and CTEs are not supported for `-ft sql/tsfile` custom queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]