This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new be04cdf326 Result issues in Build documentation, MongoDB QL, Snowflake
warehouse… (#9359)
be04cdf326 is described below
commit be04cdf326c65caa0930fe8a069cb0a6de611098
Author: Bart Maertens <[email protected]>
AuthorDate: Thu Nov 12 09:58:57 2026 +0100
Result issues in Build documentation, MongoDB QL, Snowflake warehouse…
(#9359)
* Result issues in Build documentation, MongoDB QL, Snowflake warehouse
manager and Databricks job wait, fixes #9220
* Review fixes for #9220: reject text after the MQL JSON, Fail on write
errors option, validate the Snowflake warehouse name
- MongoDB QL: parse one document or one array; trailing semicolons are
allowed, other trailing text is an error and never runs; parse positions refer
to the typed text
- MongoDB QL: write errors (also bulkWrite nErrors) fail the action only
with the new Fail on write errors option, off by default
- Build documentation and Snowflake warehouse manager: 0 errors on success
- Snowflake warehouse manager: the resolved warehouse name must be one
identifier and not empty
- Databricks job wait: keep the cancellation reason in the pt_BR message
---
.../workflow/actions/databricks-job-wait.adoc | 4 +-
.../ROOT/pages/workflow/actions/documentation.adoc | 4 +-
.../modules/ROOT/pages/workflow/actions/mql.adoc | 9 +-
.../ROOT/pages/workflow/actions/snowflake-whm.adoc | 4 +-
.../mongo-execMql/main-mongodbql-result-9220.hwf | 274 +++++++++++++++++++++
.../actions/snowflake/WarehouseManager.java | 43 +++-
.../snowflake/messages/messages_en_US.properties | 2 +
.../actions/snowflake/WarehouseManagerTest.java | 118 +++++++++
.../org/apache/hop/documentation/ActionDoc.java | 5 +
.../apache/hop/documentation/ActionDocTest.java | 77 ++++++
.../hop/databricks/client/DatabricksRunWaiter.java | 3 +-
.../databricks/ActionDatabricksJobWait.java | 6 +-
.../databricks/messages/messages_en_US.properties | 2 +-
.../databricks/messages/messages_pt_BR.properties | 4 +-
.../databricks/ActionDatabricksJobWaitTest.java | 27 ++
.../apache/hop/workflow/actions/mql/ActionMql.java | 180 +++++++++++++-
.../hop/workflow/actions/mql/ActionMqlDialog.java | 26 +-
.../actions/mql/messages/messages_en_US.properties | 11 +-
.../hop/workflow/actions/mql/ActionMqlTest.java | 181 ++++++++++++++
19 files changed, 941 insertions(+), 39 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/databricks-job-wait.adoc
b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/databricks-job-wait.adoc
index af17aeefc6..517ef80e00 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/databricks-job-wait.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/databricks-job-wait.adoc
@@ -53,7 +53,7 @@ For “submit and block on this step,” use Job Run’s built-in
**Wait for com
|Poll interval (seconds)
|Delay between status checks (default 15, at least 1)
-|Cancel Databricks run if workflow is stopped
+|Cancel Databricks run if workflow is stopped or the timeout is reached
|If checked, calls `jobs/runs/cancel` when the parent workflow stops, and also
when the timeout is reached
|Result variables
@@ -79,4 +79,4 @@ What this action passes on to the next action. See
xref:workflow/action-results.
|Result rows|Passed on unchanged.
|===
-NOTE: After every status check the action sets the result variables for the
Job ID, the Run ID, the status (the result state, like `SUCCESS` or `FAILED`,
once the run is `TERMINATED`, the life cycle state before that) and the run
page URL (the Job ID and the URL only when Databricks returns them), in its
workflow and in every parent workflow. The error variable is emptied when the
action starts and holds the error message when the action is false, except for
a canceled run: then it stays empty.
+NOTE: After every status check the action sets the result variables for the
Job ID, the Run ID, the status (the result state, like `SUCCESS` or `FAILED`,
once the run is `TERMINATED`, the life cycle state before that) and the run
page URL (the Job ID and the URL only when Databricks returns them), in its
workflow and in every parent workflow. The error variable is emptied when the
action starts and holds the error message when the action is false.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/documentation.adoc
b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/documentation.adoc
index 750174e458..ff6fe9a534 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/documentation.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/documentation.adoc
@@ -45,12 +45,10 @@ What this action passes on to the next action. See
xref:workflow/action-results.
[options="header", width="90%", cols="1,3"]
|===
|Result|Value
-|True|The documentation of the project folder was written to the target
folder. The action doesn't set its own result: it passes on the result of the
previous action, see the WARNING.
+|True|The documentation of the project folder was written to the target
folder, with 0 errors.
|False|The number of errors is 1. The target folder is empty, is a file, or
can't be written, or a file of the project can't be read (for example a
pipeline file that is not valid XML). The files written before the error stay:
the documentation is left incomplete.
|Result files|Passed on unchanged: the documentation files are not added.
|Result rows|Passed on unchanged.
|===
-WARNING: When the documentation is written, the action keeps the result of the
previous action. After a false action on an unconditional or failure hop, it is
false although the documentation was written.
-
NOTE: Files that are already in the target folder stay.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/mql.adoc
b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/mql.adoc
index 5c5e8c8f71..3fe20d36a1 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/mql.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/mql.adoc
@@ -43,7 +43,8 @@ MongoDB documentation :
https://www.mongodb.com/docs/manual/reference/command/
|Use external file|Enable this option to load the JSON instructions from a
file.
|Filename|The filename for the file with JSON instructions.
|Use variable substitution|Enables variables to be used in the Script.
-|JSON|The JSON instructions to execute: one JSON document, or several in a
JSON array (`[ {...}, {...} ]`).
+|Fail on write errors|Make the action false when MongoDB reports write errors
in its answer (`writeErrors`, `writeConcernError` or, for `bulkWrite`,
`nErrors`), for example an `insert` with a duplicate `_id`. Leave it unchecked
to accept a partial write, for example an unordered insert (`"ordered": false`)
that skips duplicate keys: the write errors are then written to the log.
Unchecked by default.
+|JSON|The JSON instructions to execute: one JSON document, or several in a
JSON array (`[ {...}, {...} ]`). Semicolons at the end are allowed. Any other
text after the document or the array is an error.
|===
== Examples
@@ -71,11 +72,11 @@ What this action passes on to the next action. See
xref:workflow/action-results.
|===
|Result|Value
|True|Every instruction ran, with 0 errors. An instruction that returns
documents, like `find`, is true too: the documents are not passed on.
-|False|The number of errors is 1. *MongoDB Connection* is empty or doesn't
exist, the server can't be reached or refuses the login, *Use external file* is
checked while *Filename* is empty or the file can't be read, the JSON is empty
or not valid, or MongoDB refuses an instruction (for example an unknown
command). The instructions of an array run one after the other: the ones before
the failing instruction stay done, the ones after it are not run.
+|False|The number of errors is 1. *MongoDB Connection* is empty or doesn't
exist, the server can't be reached or refuses the login, *Use external file* is
checked while *Filename* is empty or the file can't be read, the JSON is empty
or not valid, MongoDB refuses an instruction (for example an unknown command),
or *Fail on write errors* is checked and MongoDB reports write errors or a
write concern error in its answer (for example an `insert` with a duplicate
`_id`). The instructions of [...]
|Result files|Passed on unchanged.
|Result rows|Passed on unchanged.
|===
-NOTE: Several instructions must be in a JSON array. Instructions separated by
commas without the brackets are not an error, but only the first one runs.
+NOTE: When an instruction has write errors, MongoDB still writes the documents
before the failing one (and, for an unordered write, the documents after it).
With *Fail on write errors* checked, the action is false and the instructions
after it are not run.
-NOTE: Write errors that MongoDB reports in its answer don't make the action
false. For example an `insert` with a duplicate `_id` is true, although the
duplicate document is not inserted.
+NOTE: Up to Hop 2.20, text after the first JSON document or array was ignored:
in `{"drop": "a"}, {"drop": "b"}` only the first instruction ran. Since Hop
2.21 that text makes the action false, and none of the instructions run. Put
several instructions in a JSON array.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/snowflake-whm.adoc
b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/snowflake-whm.adoc
index 4fe89468fc..ef365fcbda 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/snowflake-whm.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/snowflake-whm.adoc
@@ -105,9 +105,7 @@ What this action passes on to the next action. See
xref:workflow/action-results.
|===
|Result|Value
|True|The statement ran, with 0 errors.
-|False|The action doesn't count errors: the number of errors stays 0.
*Connection* is empty, no *Action* is selected, for *Create warehouse* *Max
cluster size* is not a positive number or *Min cluster size* or *Auto suspend
(minutes)* is negative or not a number, the connection doesn't exist or can't
connect, or Snowflake refuses the statement, for example when the warehouse
exists and *Fail if warehouse exists* is checked, or doesn't exist and *Fail if
warehouse does not exist* is checked.
+|False|The number of errors is 1. *Connection* is empty, no *Action* is
selected, *Warehouse name* is empty or, with its variables resolved, is not one
Snowflake identifier (letters, digits, `_` and `$`, not starting with a digit,
or a name in double quotes), for *Create warehouse* *Max cluster size* is not a
positive number or *Min cluster size* or *Auto suspend (minutes)* is negative
or not a number, the connection doesn't exist or can't connect, or Snowflake
refuses the statement, for [...]
|Result files|Passed on unchanged.
|Result rows|Passed on unchanged.
|===
-
-NOTE: *Create warehouse* and *Alter warehouse* use *Warehouse name* as it is
typed: variables in the name are not resolved, unlike for *Drop warehouse*,
*Resume warehouse* and *Suspend warehouse*.
diff --git
a/integration-tests/mongo/tests/mongo-execMql/main-mongodbql-result-9220.hwf
b/integration-tests/mongo/tests/mongo-execMql/main-mongodbql-result-9220.hwf
new file mode 100644
index 0000000000..85da9eeb68
--- /dev/null
+++ b/integration-tests/mongo/tests/mongo-execMql/main-mongodbql-result-9220.hwf
@@ -0,0 +1,274 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<workflow>
+ <name>main-mongodbql-result-9220</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>MongoDB QL: write errors fail only with Fail on write errors,
text after the instruction is an error (#9220)</description>
+ <extended_description/>
+ <created_user>-</created_user>
+ <modified_user>-</modified_user>
+ <created_date>2026/11/07 12:00:00.000</created_date>
+ <modified_date>2026/11/11 12:00:00.000</modified_date>
+ <workflow_version/>
+ <parameters>
+ <parameter>
+ <name>MONGO_CONNECTION</name>
+ <description/>
+ <default_value>mongo</default_value>
+ </parameter>
+ </parameters>
+ <actions>
+ <action>
+ <repeat>N</repeat>
+ <schedulerType>0</schedulerType>
+ <intervalSeconds>0</intervalSeconds>
+ <intervalMinutes>60</intervalMinutes>
+ <DayOfMonth>1</DayOfMonth>
+ <weekDay>1</weekDay>
+ <minutes>0</minutes>
+ <hour>12</hour>
+ <doNotWaitOnFirstExecution>N</doNotWaitOnFirstExecution>
+ <name>Start</name>
+ <description/>
+ <type>SPECIAL</type>
+ <attributes/>
+ <xloc>48</xloc>
+ <yloc>48</yloc>
+ <parallel>N</parallel>
+ <attributes_hac/>
+ </action>
+ <action>
+ <mql>{"delete": "mql_9220", "deletes": [{"q": {}, "limit": 0}]}</mql>
+ <connection>${MONGO_CONNECTION}</connection>
+ <useExternalFile>N</useExternalFile>
+ <commandFile/>
+ <useVariableSubstitution>N</useVariableSubstitution>
+ <failOnWriteErrors>N</failOnWriteErrors>
+ <name>Clear collection</name>
+ <description/>
+ <type>MQL</type>
+ <attributes/>
+ <xloc>208</xloc>
+ <yloc>48</yloc>
+ <parallel>N</parallel>
+ <attributes_hac/>
+ </action>
+ <action>
+ <mql>{"insert": "mql_9220", "documents": [{"_id": 1}, {"_id": 1},
{"_id": 2}], "ordered": false}</mql>
+ <connection>${MONGO_CONNECTION}</connection>
+ <useExternalFile>N</useExternalFile>
+ <commandFile/>
+ <useVariableSubstitution>N</useVariableSubstitution>
+ <failOnWriteErrors>N</failOnWriteErrors>
+ <name>Unordered duplicate is true</name>
+ <description/>
+ <type>MQL</type>
+ <attributes/>
+ <xloc>400</xloc>
+ <yloc>48</yloc>
+ <parallel>N</parallel>
+ <attributes_hac/>
+ </action>
+ <action>
+ <mql>{"insert": "mql_9220", "documents": [{"_id": 2}]}</mql>
+ <connection>${MONGO_CONNECTION}</connection>
+ <useExternalFile>N</useExternalFile>
+ <commandFile/>
+ <useVariableSubstitution>N</useVariableSubstitution>
+ <failOnWriteErrors>Y</failOnWriteErrors>
+ <name>Duplicate _id fails on write errors</name>
+ <description/>
+ <type>MQL</type>
+ <attributes/>
+ <xloc>592</xloc>
+ <yloc>48</yloc>
+ <parallel>N</parallel>
+ <attributes_hac/>
+ </action>
+ <action>
+ <mql>{"insert": "mql_9220", "documents": [{"_id": 3}]},
+{"insert": "mql_9220", "documents": [{"_id": 4}]}</mql>
+ <connection>${MONGO_CONNECTION}</connection>
+ <useExternalFile>N</useExternalFile>
+ <commandFile/>
+ <useVariableSubstitution>N</useVariableSubstitution>
+ <failOnWriteErrors>N</failOnWriteErrors>
+ <name>Comma-separated is false</name>
+ <description/>
+ <type>MQL</type>
+ <attributes/>
+ <xloc>784</xloc>
+ <yloc>48</yloc>
+ <parallel>N</parallel>
+ <attributes_hac/>
+ </action>
+ <action>
+ <mql>[
+ {"insert": "mql_9220", "documents": [{"_id": 3}]},
+ {"insert": "mql_9220", "documents": [{"_id": 4}]}
+];</mql>
+ <connection>${MONGO_CONNECTION}</connection>
+ <useExternalFile>N</useExternalFile>
+ <commandFile/>
+ <useVariableSubstitution>N</useVariableSubstitution>
+ <failOnWriteErrors>Y</failOnWriteErrors>
+ <name>Nothing was inserted</name>
+ <description/>
+ <type>MQL</type>
+ <attributes/>
+ <xloc>976</xloc>
+ <yloc>48</yloc>
+ <parallel>N</parallel>
+ <attributes_hac/>
+ </action>
+ <action>
+ <mql>{"drop": "mql_9220"}</mql>
+ <connection>${MONGO_CONNECTION}</connection>
+ <useExternalFile>N</useExternalFile>
+ <commandFile/>
+ <useVariableSubstitution>N</useVariableSubstitution>
+ <failOnWriteErrors>N</failOnWriteErrors>
+ <name>Drop collection</name>
+ <description/>
+ <type>MQL</type>
+ <attributes/>
+ <xloc>1168</xloc>
+ <yloc>48</yloc>
+ <parallel>N</parallel>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Success</name>
+ <description/>
+ <type>SUCCESS</type>
+ <attributes/>
+ <xloc>1360</xloc>
+ <yloc>48</yloc>
+ <parallel>N</parallel>
+ <attributes_hac/>
+ </action>
+ <action>
+ <always_log_rows>N</always_log_rows>
+ <name>Abort workflow</name>
+ <description/>
+ <type>ABORT</type>
+ <attributes/>
+ <xloc>784</xloc>
+ <yloc>192</yloc>
+ <parallel>N</parallel>
+ <attributes_hac/>
+ </action>
+ </actions>
+ <hops>
+ <hop>
+ <from>Start</from>
+ <to>Clear collection</to>
+ <evaluation>Y</evaluation>
+ <unconditional>Y</unconditional>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Clear collection</from>
+ <to>Unordered duplicate is true</to>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Clear collection</from>
+ <to>Abort workflow</to>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Unordered duplicate is true</from>
+ <to>Duplicate _id fails on write errors</to>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Unordered duplicate is true</from>
+ <to>Abort workflow</to>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Duplicate _id fails on write errors</from>
+ <to>Comma-separated is false</to>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Duplicate _id fails on write errors</from>
+ <to>Abort workflow</to>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Comma-separated is false</from>
+ <to>Nothing was inserted</to>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Comma-separated is false</from>
+ <to>Abort workflow</to>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Nothing was inserted</from>
+ <to>Drop collection</to>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Nothing was inserted</from>
+ <to>Abort workflow</to>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Drop collection</from>
+ <to>Success</to>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Drop collection</from>
+ <to>Abort workflow</to>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ <enabled>Y</enabled>
+ </hop>
+ </hops>
+ <notepads/>
+ <attributes/>
+</workflow>
diff --git
a/plugins/databases/snowflake/src/main/java/org/apache/hop/workflow/actions/snowflake/WarehouseManager.java
b/plugins/databases/snowflake/src/main/java/org/apache/hop/workflow/actions/snowflake/WarehouseManager.java
index e0ff861075..0fa707d7da 100644
---
a/plugins/databases/snowflake/src/main/java/org/apache/hop/workflow/actions/snowflake/WarehouseManager.java
+++
b/plugins/databases/snowflake/src/main/java/org/apache/hop/workflow/actions/snowflake/WarehouseManager.java
@@ -22,6 +22,7 @@ import static
org.apache.hop.workflow.action.validator.ActionValidatorUtils.notB
import static
org.apache.hop.workflow.action.validator.AndValidator.putValidators;
import java.util.List;
+import java.util.regex.Pattern;
import lombok.Getter;
import lombok.Setter;
import org.apache.hop.core.Const;
@@ -58,6 +59,8 @@ public class WarehouseManager extends ActionBase implements
Cloneable, IAction {
public static final String REPLACE = "replace";
public static final String FAIL_IF_EXISTS = "failIfExists";
public static final String WAREHOUSE_NAME = "warehouseName";
+ private static final Pattern VALID_IDENTIFIER =
+ Pattern.compile("[A-Za-z_][A-Za-z0-9_$]*|\"(?:[^\"]|\"\")+\"");
public static final String WAREHOUSE_SIZE = "warehouseSize";
public static final String WAREHOUSE_TYPE = "warehouseType";
public static final String MAX_CLUSTER_COUNT = "maxClusterCount";
@@ -267,6 +270,17 @@ public class WarehouseManager extends ActionBase
implements Cloneable, IAction {
} else if (Utils.isEmpty(managementAction)) {
logError(BaseMessages.getString(PKG,
"SnowflakeWarehouseManager.Validate.ManagementAction"));
result = false;
+ } else if (Utils.isEmpty(resolve(warehouseName))) {
+ logError(
+ BaseMessages.getString(PKG,
"SnowflakeWarehouseManager.Validate.WarehouseNameEmpty"));
+ result = false;
+ } else if (!isValidIdentifier(resolve(warehouseName))) {
+ logError(
+ BaseMessages.getString(
+ PKG,
+ "SnowflakeWarehouseManager.Validate.WarehouseNameInvalid",
+ resolve(warehouseName)));
+ result = false;
} else if
(managementAction.equals(MANAGEMENT_ACTIONS[MANAGEMENT_ACTION_CREATE])) {
if (!Utils.isEmpty(resolve(maxClusterCount))
&& Const.toInt(resolve(maxClusterCount), -1) <= 0) {
@@ -300,11 +314,24 @@ public class WarehouseManager extends ActionBase
implements Cloneable, IAction {
return result;
}
+ /**
+ * The warehouse name goes into the SQL as it is, so it must be one
identifier: unquoted (a letter
+ * or underscore followed by letters, digits, underscores or dollar signs)
or double-quoted (with
+ * a double quote in it written twice).
+ *
+ * @param name the resolved warehouse name
+ * @return true when the name is a valid Snowflake identifier
+ */
+ static boolean isValidIdentifier(String name) {
+ return name != null && VALID_IDENTIFIER.matcher(name).matches();
+ }
+
public Result execute(Result previousResult, int nr) throws HopException {
Result result = previousResult;
result.setResult(validate());
if (!result.isResult()) {
+ result.setNrErrors(1);
return result;
}
DatabaseMeta databaseMeta = parentWorkflowMeta.findDatabase(connection,
getVariables());
@@ -340,16 +367,18 @@ public class WarehouseManager extends ActionBase
implements Cloneable, IAction {
logDebug("Executing SQL " + sql);
db.execStatements(sql);
logBasic(successMessage);
+ result.setNrErrors(0);
} catch (Exception ex) {
logError("Error managing warehouse", ex);
result.setResult(false);
+ result.setNrErrors(1);
}
return result;
}
- private String getDropSQL() {
+ String getDropSQL() {
StringBuilder sql = new StringBuilder();
sql.append("DROP WAREHOUSE ");
if (!failIfNotExists) {
@@ -359,7 +388,7 @@ public class WarehouseManager extends ActionBase implements
Cloneable, IAction {
return sql.toString();
}
- private String getResumeSQL() {
+ String getResumeSQL() {
StringBuilder sql = new StringBuilder();
sql.append(CONST_ALTER_WAREHOUSE);
if (!failIfNotExists) {
@@ -369,7 +398,7 @@ public class WarehouseManager extends ActionBase implements
Cloneable, IAction {
return sql.toString();
}
- private String getSuspendSQL() {
+ String getSuspendSQL() {
StringBuilder sql = new StringBuilder();
sql.append(CONST_ALTER_WAREHOUSE);
if (!failIfNotExists) {
@@ -379,7 +408,7 @@ public class WarehouseManager extends ActionBase implements
Cloneable, IAction {
return sql.toString();
}
- private String getCreateSQL() {
+ String getCreateSQL() {
StringBuilder sql = new StringBuilder();
sql.append("CREATE ");
if (replace) {
@@ -389,7 +418,7 @@ public class WarehouseManager extends ActionBase implements
Cloneable, IAction {
if (!failIfExists && !replace) {
sql.append("IF NOT EXISTS ");
}
- sql.append(warehouseName).append(" WITH ");
+ sql.append(resolve(warehouseName)).append(" WITH ");
if (!StringUtil.isEmpty(resolve(warehouseSize))) {
sql.append("WAREHOUSE_SIZE =
'").append(resolve(warehouseSize)).append("' ");
@@ -426,13 +455,13 @@ public class WarehouseManager extends ActionBase
implements Cloneable, IAction {
return sql.toString();
}
- private String getAlterSQL() {
+ String getAlterSQL() {
StringBuilder sql = new StringBuilder();
sql.append(CONST_ALTER_WAREHOUSE);
if (!failIfNotExists) {
sql.append(CONST_IF_EXISTS);
}
- sql.append(warehouseName).append(" SET ");
+ sql.append(resolve(warehouseName)).append(" SET ");
if (!StringUtil.isEmpty(resolve(warehouseSize))) {
sql.append("WAREHOUSE_SIZE =
'").append(resolve(warehouseSize)).append("' ");
diff --git
a/plugins/databases/snowflake/src/main/resources/org/apache/hop/workflow/actions/snowflake/messages/messages_en_US.properties
b/plugins/databases/snowflake/src/main/resources/org/apache/hop/workflow/actions/snowflake/messages/messages_en_US.properties
index 949b3b1731..b21bcf1d8c 100644
---
a/plugins/databases/snowflake/src/main/resources/org/apache/hop/workflow/actions/snowflake/messages/messages_en_US.properties
+++
b/plugins/databases/snowflake/src/main/resources/org/apache/hop/workflow/actions/snowflake/messages/messages_en_US.properties
@@ -79,3 +79,5 @@ SnowflakeWarehouseManager.Validate.DatabaseIsEmpty=No
database connection specif
SnowflakeWarehouseManager.Validate.ManagementAction=No action specified
SnowflakeWarehouseManager.Validate.MaxClusterCount=Max cluster count {0} is
not valid
SnowflakeWarehouseManager.Validate.MinClusterCount=Min cluster count {0} is
not valid
+SnowflakeWarehouseManager.Validate.WarehouseNameEmpty=No warehouse name
specified
+SnowflakeWarehouseManager.Validate.WarehouseNameInvalid=Warehouse name {0} is
not a valid Snowflake identifier: use letters, digits, _ and $, or put the name
in double quotes
diff --git
a/plugins/databases/snowflake/src/test/java/org/apache/hop/workflow/actions/snowflake/WarehouseManagerTest.java
b/plugins/databases/snowflake/src/test/java/org/apache/hop/workflow/actions/snowflake/WarehouseManagerTest.java
new file mode 100644
index 0000000000..8c12245233
--- /dev/null
+++
b/plugins/databases/snowflake/src/test/java/org/apache/hop/workflow/actions/snowflake/WarehouseManagerTest.java
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.workflow.actions.snowflake;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import org.apache.hop.core.Result;
+import org.apache.hop.junit.rules.RestoreHopEngineEnvironmentExtension;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+class WarehouseManagerTest {
+
+ @RegisterExtension
+ static RestoreHopEngineEnvironmentExtension env = new
RestoreHopEngineEnvironmentExtension();
+
+ private WarehouseManager newAction(int managementAction) {
+ WarehouseManager action = new WarehouseManager("whm");
+ action.setManagementActionById(managementAction);
+ action.setVariable("WAREHOUSE", "HOP_WH");
+ action.setWarehouseName("${WAREHOUSE}");
+ return action;
+ }
+
+ @Test
+ void warehouseNameIsResolvedForEveryAction() {
+ assertTrue(
+ newAction(WarehouseManager.MANAGEMENT_ACTION_CREATE)
+ .getCreateSQL()
+ .contains(" HOP_WH WITH "));
+ assertTrue(
+
newAction(WarehouseManager.MANAGEMENT_ACTION_ALTER).getAlterSQL().contains("
HOP_WH SET "));
+ assertTrue(
+
newAction(WarehouseManager.MANAGEMENT_ACTION_DROP).getDropSQL().contains("
HOP_WH;"));
+ assertTrue(
+ newAction(WarehouseManager.MANAGEMENT_ACTION_RESUME)
+ .getResumeSQL()
+ .contains(" HOP_WH RESUME;"));
+ assertTrue(
+ newAction(WarehouseManager.MANAGEMENT_ACTION_SUSPEND)
+ .getSuspendSQL()
+ .contains(" HOP_WH SUSPEND;"));
+ }
+
+ @Test
+ void warehouseNameMustBeOneIdentifier() {
+ WarehouseManager action =
newAction(WarehouseManager.MANAGEMENT_ACTION_CREATE);
+ action.setConnection("snowflake");
+ assertTrue(action.validate());
+
+ // Quoted names can hold any character
+ action.setVariable("WAREHOUSE", "\"My \"\"Warehouse\"\"; 1\"");
+ assertTrue(action.validate());
+ action.setVariable("WAREHOUSE", "_WH$1");
+ assertTrue(action.validate());
+
+ // Empty after resolving
+ action.setVariable("WAREHOUSE", "");
+ assertFalse(action.validate());
+ action.setWarehouseName(null);
+ assertFalse(action.validate());
+
+ // Extra statements, spaces or an unbalanced quote are refused
+ action.setWarehouseName("${WAREHOUSE}");
+ action.setVariable("WAREHOUSE", "X WITH WAREHOUSE_SIZE='XSMALL'; DROP
DATABASE prod; --");
+ assertFalse(action.validate());
+ action.setVariable("WAREHOUSE", "MY WH");
+ assertFalse(action.validate());
+ action.setVariable("WAREHOUSE", "\"a\"; DROP DATABASE prod; --\"");
+ assertFalse(action.validate());
+ action.setVariable("WAREHOUSE", "1WH");
+ assertFalse(action.validate());
+ }
+
+ @Test
+ void failedValidationCountsOneError() throws Exception {
+ // No connection: validate() fails
+ WarehouseManager action =
newAction(WarehouseManager.MANAGEMENT_ACTION_RESUME);
+
+ Result result = new Result();
+ action.execute(result, 0);
+
+ assertFalse(result.isResult());
+ assertEquals(1, result.getNrErrors());
+ }
+
+ @Test
+ void failedStatementCountsOneError() throws Exception {
+ // The connection doesn't exist in the workflow: the statement can't run
+ WarehouseManager action =
newAction(WarehouseManager.MANAGEMENT_ACTION_RESUME);
+ action.setConnection("missing");
+ action.setParentWorkflowMeta(new WorkflowMeta());
+
+ Result result = new Result();
+ action.execute(result, 0);
+
+ assertFalse(result.isResult());
+ assertEquals(1, result.getNrErrors());
+ }
+}
diff --git
a/plugins/misc/documentation/src/main/java/org/apache/hop/documentation/ActionDoc.java
b/plugins/misc/documentation/src/main/java/org/apache/hop/documentation/ActionDoc.java
index 5ef0bb376d..a4d48ff8fa 100644
---
a/plugins/misc/documentation/src/main/java/org/apache/hop/documentation/ActionDoc.java
+++
b/plugins/misc/documentation/src/main/java/org/apache/hop/documentation/ActionDoc.java
@@ -143,6 +143,11 @@ public class ActionDoc extends ActionBase implements
Cloneable, IAction {
includingMetadata,
generateHtml,
removeMarkdown);
+ // The result of this action is its own: true when the documentation is
written.
+ // DocBuilder sets it back to false (and 1 error) when that fails.
+ //
+ prevResult.setResult(true);
+ prevResult.setNrErrors(0);
docBuilder.buildDocumentation(prevResult);
return prevResult;
}
diff --git
a/plugins/misc/documentation/src/test/java/org/apache/hop/documentation/ActionDocTest.java
b/plugins/misc/documentation/src/test/java/org/apache/hop/documentation/ActionDocTest.java
new file mode 100644
index 0000000000..d8a297ba6c
--- /dev/null
+++
b/plugins/misc/documentation/src/test/java/org/apache/hop/documentation/ActionDocTest.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.documentation;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.apache.hop.core.Result;
+import org.apache.hop.junit.rules.RestoreHopEngineEnvironmentExtension;
+import org.apache.hop.metadata.serializer.memory.MemoryMetadataProvider;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+
+class ActionDocTest {
+
+ @RegisterExtension
+ static RestoreHopEngineEnvironmentExtension env = new
RestoreHopEngineEnvironmentExtension();
+
+ @TempDir Path projectHome;
+ @TempDir Path target;
+
+ private ActionDoc newAction(String targetFolder) {
+ ActionDoc action = new ActionDoc();
+ action.setName("doc");
+ action.setMetadataProvider(new MemoryMetadataProvider());
+ action.setVariable(DocBuilder.VAR_PROJECT_NAME, "test");
+ action.setVariable(DocBuilder.VAR_PROJECT_HOME, projectHome.toString());
+ action.setTargetParentFolder(targetFolder);
+ action.setIncludingMetadata(false);
+ return action;
+ }
+
+ @Test
+ void writtenDocumentationIsTrueAfterAFalseAction() throws Exception {
+ Files.writeString(projectHome.resolve("readme.txt"), "not documented");
+
+ // The previous action was false, for example on an unconditional or a
failure hop
+ Result previous = new Result();
+ previous.setResult(false);
+ previous.setNrErrors(1);
+
+ Result result = newAction(target.toString()).execute(previous, 0);
+
+ assertTrue(result.isResult());
+ assertEquals(0, result.getNrErrors());
+ assertTrue(Files.exists(target.resolve(DocBuilder.INDEX_MD)));
+ }
+
+ @Test
+ void targetThatIsAFileIsFalse() throws Exception {
+ Path file = Files.writeString(target.resolve("a-file"), "x");
+
+ Result result = newAction(file.toString()).execute(new Result(), 0);
+
+ assertFalse(result.isResult());
+ assertEquals(1, result.getNrErrors());
+ }
+}
diff --git
a/plugins/tech/databricks/src/main/java/org/apache/hop/databricks/client/DatabricksRunWaiter.java
b/plugins/tech/databricks/src/main/java/org/apache/hop/databricks/client/DatabricksRunWaiter.java
index d5de7df9b7..d5033c4496 100644
---
a/plugins/tech/databricks/src/main/java/org/apache/hop/databricks/client/DatabricksRunWaiter.java
+++
b/plugins/tech/databricks/src/main/java/org/apache/hop/databricks/client/DatabricksRunWaiter.java
@@ -47,7 +47,8 @@ public final class DatabricksRunWaiter {
/**
* @param timeoutSec 0 or negative = no timeout
* @param pollSec minimum 1 second between polls
- * @param cancelOnStop if true, call cancelRun when stopped or on timeout
+ * @param cancelOnStop if true, call cancelRun when the workflow is stopped
and when the timeout
+ * is reached
*/
public static DatabricksRunStatus waitFor(
DatabricksJobsClient client,
diff --git
a/plugins/tech/databricks/src/main/java/org/apache/hop/workflow/actions/databricks/ActionDatabricksJobWait.java
b/plugins/tech/databricks/src/main/java/org/apache/hop/workflow/actions/databricks/ActionDatabricksJobWait.java
index 7a952af05a..d688ffeba8 100644
---
a/plugins/tech/databricks/src/main/java/org/apache/hop/workflow/actions/databricks/ActionDatabricksJobWait.java
+++
b/plugins/tech/databricks/src/main/java/org/apache/hop/workflow/actions/databricks/ActionDatabricksJobWait.java
@@ -184,11 +184,13 @@ public class ActionDatabricksJobWait extends ActionBase
implements Cloneable, IA
&& "CANCELED".equalsIgnoreCase(status.getResultState())) {
result.setNrErrors(1);
result.setResult(false);
- logError(
+ String msg =
BaseMessages.getString(
PKG,
"ActionDatabricksJobWait.Error.Canceled",
- Const.NVL(status.getStateMessage(), "")));
+ Const.NVL(status.getStateMessage(), ""));
+ setResultVariable(resultVariableError, msg);
+ logError(msg);
} else {
result.setNrErrors(1);
result.setResult(false);
diff --git
a/plugins/tech/databricks/src/main/resources/org/apache/hop/workflow/actions/databricks/messages/messages_en_US.properties
b/plugins/tech/databricks/src/main/resources/org/apache/hop/workflow/actions/databricks/messages/messages_en_US.properties
index 55e563b327..65fbd02f2e 100644
---
a/plugins/tech/databricks/src/main/resources/org/apache/hop/workflow/actions/databricks/messages/messages_en_US.properties
+++
b/plugins/tech/databricks/src/main/resources/org/apache/hop/workflow/actions/databricks/messages/messages_en_US.properties
@@ -101,7 +101,7 @@ ActionDatabricksJobWait.Connection.Tooltip=Workspace
connection (Jobs API / PAT)
ActionDatabricksJobWait.RunId.Label=Run ID
ActionDatabricksJobWait.Timeout.Label=Timeout (seconds, 0=none)
ActionDatabricksJobWait.Poll.Label=Poll interval (seconds)
-ActionDatabricksJobWait.CancelOnStop.Label=Cancel Databricks run if workflow
is stopped
+ActionDatabricksJobWait.CancelOnStop.Label=Cancel Databricks run if workflow
is stopped or the timeout is reached
ActionDatabricksJobWait.VarJobId.Label=Result variable: Job ID
ActionDatabricksJobWait.VarRunId.Label=Result variable: Run ID
ActionDatabricksJobWait.VarStatus.Label=Result variable: Status
diff --git
a/plugins/tech/databricks/src/main/resources/org/apache/hop/workflow/actions/databricks/messages/messages_pt_BR.properties
b/plugins/tech/databricks/src/main/resources/org/apache/hop/workflow/actions/databricks/messages/messages_pt_BR.properties
index e266aa36a7..0960609294 100644
---
a/plugins/tech/databricks/src/main/resources/org/apache/hop/workflow/actions/databricks/messages/messages_pt_BR.properties
+++
b/plugins/tech/databricks/src/main/resources/org/apache/hop/workflow/actions/databricks/messages/messages_pt_BR.properties
@@ -102,7 +102,7 @@ ActionDatabricksJobWait.Connection.Tooltip=Conexão de
espaço de trabalho (Jobs
ActionDatabricksJobWait.RunId.Label=Correr ID
ActionDatabricksJobWait.Timeout.Label=Tempo limite (segundos, 0=nenhum)
ActionDatabricksJobWait.Poll.Label=Intervalo de pesquisa (segundos)
-ActionDatabricksJobWait.CancelOnStop.Label=Cancele a execução de Databricks se
o fluxo de trabalho for interrompido
+ActionDatabricksJobWait.CancelOnStop.Label=Cancele a execução de Databricks se
o fluxo de trabalho for interrompido ou o tempo limite for atingido
ActionDatabricksJobWait.VarJobId.Label=Variável do resultado: ID do trabalho
ActionDatabricksJobWait.VarRunId.Label=Variável de resultado: ID de execução
ActionDatabricksJobWait.VarStatus.Label=Variável do resultado: Status
@@ -112,5 +112,5 @@ ActionDatabricksJobWait.Error.NoConnection=Por favor,
selecione uma conexão Dat
ActionDatabricksJobWait.Error.NoMetadataProvider=Nenhum provedor de metadados
disponível.
ActionDatabricksJobWait.Error.ConnectionNotFound=Conexão de databricks não
encontrada:
ActionDatabricksJobWait.Error.NoRunId=O ID de execução é necessário (por
exemplo, variável DatabricksRunId de Job Run)
-ActionDatabricksJobWait.Error.Canceled=Databricks foi cancelado.
+ActionDatabricksJobWait.Error.Canceled=A execução de Databricks foi cancelada:
{0}
ActionDatabricksJobWait.Error.Execute=Erro em esperar que Databricks corra.
diff --git
a/plugins/tech/databricks/src/test/java/org/apache/hop/workflow/actions/databricks/ActionDatabricksJobWaitTest.java
b/plugins/tech/databricks/src/test/java/org/apache/hop/workflow/actions/databricks/ActionDatabricksJobWaitTest.java
index a769214d11..836a963c59 100644
---
a/plugins/tech/databricks/src/test/java/org/apache/hop/workflow/actions/databricks/ActionDatabricksJobWaitTest.java
+++
b/plugins/tech/databricks/src/test/java/org/apache/hop/workflow/actions/databricks/ActionDatabricksJobWaitTest.java
@@ -108,4 +108,31 @@ class ActionDatabricksJobWaitTest {
assertFalse(result.getResult());
assertEquals("FAILED", action.getVariable("DatabricksStatus"));
}
+
+ @Test
+ void canceledRunSetsErrorVariable() throws Exception {
+ when(client.getRun(anyLong()))
+ .thenReturn(
+ new DatabricksRunStatus(
+ 1L,
+ 2L,
+ DatabricksRunLifeCycleState.TERMINATED,
+ "CANCELED",
+ "Run cancelled by user",
+ null));
+
+ ActionDatabricksJobWait action = new ActionDatabricksJobWait("wait");
+ action.setMetadataProvider(metadataProvider);
+ action.setConnectionName("dbx");
+ action.setRunId("1");
+ action.setClientFactory((c, v) -> client);
+
+ Result result = new Result();
+ action.execute(result, 0);
+ assertFalse(result.getResult());
+ assertEquals(1, result.getNrErrors());
+ assertEquals("CANCELED", action.getVariable("DatabricksStatus"));
+ String error = action.getVariable("DatabricksError");
+ assertTrue(error != null && error.contains("Run cancelled by user"),
error);
+ }
}
diff --git
a/plugins/tech/mongodb/src/main/java/org/apache/hop/workflow/actions/mql/ActionMql.java
b/plugins/tech/mongodb/src/main/java/org/apache/hop/workflow/actions/mql/ActionMql.java
index a9e5927530..b3e7c8ccc8 100644
---
a/plugins/tech/mongodb/src/main/java/org/apache/hop/workflow/actions/mql/ActionMql.java
+++
b/plugins/tech/mongodb/src/main/java/org/apache/hop/workflow/actions/mql/ActionMql.java
@@ -17,6 +17,7 @@
package org.apache.hop.workflow.actions.mql;
+import java.util.ArrayList;
import java.util.List;
import org.apache.hop.core.ICheckResult;
import org.apache.hop.core.Result;
@@ -28,6 +29,7 @@ import org.apache.hop.core.variables.IVariables;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.metadata.api.HopMetadataProperty;
import org.apache.hop.metadata.api.IHopMetadataProvider;
+import org.apache.hop.mongo.MongoDbException;
import org.apache.hop.mongo.metadata.MongoDbConnection;
import org.apache.hop.mongo.wrapper.MongoClientWrapper;
import org.apache.hop.resource.ResourceEntry;
@@ -38,7 +40,12 @@ import org.apache.hop.workflow.action.ActionBase;
import org.apache.hop.workflow.action.IAction;
import org.apache.hop.workflow.action.validator.ActionValidatorUtils;
import org.apache.hop.workflow.action.validator.AndValidator;
+import org.bson.BsonType;
import org.bson.Document;
+import org.bson.codecs.DecoderContext;
+import org.bson.codecs.DocumentCodec;
+import org.bson.json.JsonParseException;
+import org.bson.json.JsonReader;
/** This defines a MongoDB QL action. */
@Action(
@@ -68,6 +75,9 @@ public class ActionMql extends ActionBase implements
Cloneable, IAction {
@HopMetadataProperty(key = "useVariableSubstitution")
private boolean useVariableSubstitution = true;
+ @HopMetadataProperty(key = "failOnWriteErrors")
+ private boolean failOnWriteErrors = false;
+
public ActionMql(String n) {
super(n, "");
mql = null;
@@ -118,6 +128,14 @@ public class ActionMql extends ActionBase implements
Cloneable, IAction {
useVariableSubstitution = subst;
}
+ public boolean isFailOnWriteErrors() {
+ return failOnWriteErrors;
+ }
+
+ public void setFailOnWriteErrors(boolean failOnWriteErrors) {
+ this.failOnWriteErrors = failOnWriteErrors;
+ }
+
@Override
public Result execute(Result result, int nr) {
@@ -161,23 +179,22 @@ public class ActionMql extends ActionBase implements
Cloneable, IAction {
mqlToExecute = resolve(mqlToExecute);
}
- final String finalMql = mqlToExecute;
+ final List<Document> commands = parseCommands(mqlToExecute);
wrapper.perform(
dbName,
db -> {
- String trimmed = finalMql.trim();
- // Multiple instructions
- if (trimmed.startsWith("[")) {
- List<Document> commands =
- org.bson.Document.parse("{ \"array\": " + trimmed + " }")
- .getList("array", Document.class);
- for (Document cmd : commands) {
- db.runCommand(cmd);
+ for (int i = 0; i < commands.size(); i++) {
+ Document reply = db.runCommand(commands.get(i));
+ try {
+ checkReply(reply, i + 1, commands.size());
+ } catch (MongoDbException e) {
+ if (failOnWriteErrors) {
+ throw e;
+ }
+ logBasic(
+ BaseMessages.getString(PKG,
"ActionMQL.WriteErrorsIgnored", e.getMessage()));
}
- } else {
- // One instruction
- db.runCommand(Document.parse(trimmed));
}
return null;
});
@@ -201,6 +218,145 @@ public class ActionMql extends ActionBase implements
Cloneable, IAction {
return result;
}
+ /**
+ * Parse the commands to run: one JSON document, or a JSON array with one
document per command.
+ * Semicolons and whitespace at the end are allowed. Any other text after
the document or the
+ * array is an error: it is neither run nor silently ignored.
+ *
+ * <p>The text is parsed as it is, so a position in a JSON error matches the
text in the dialog.
+ *
+ * @param mqlToExecute the JSON text
+ * @return the commands, in the order they are to be run
+ * @throws HopException when there is no command or the text is not valid
+ */
+ static List<Document> parseCommands(String mqlToExecute) throws HopException
{
+ String text = mqlToExecute == null ? "" : mqlToExecute;
+ int end = text.length();
+ while (end > 0
+ && (Character.isWhitespace(text.charAt(end - 1)) || text.charAt(end -
1) == ';')) {
+ end--;
+ }
+ text = text.substring(0, end);
+ if (text.isBlank()) {
+ throw new HopException(BaseMessages.getString(PKG,
"ActionMQL.NoCommandFound"));
+ }
+
+ List<Document> commands = new ArrayList<>();
+ DocumentCodec codec = new DocumentCodec();
+ DecoderContext context = DecoderContext.builder().build();
+ try (JsonReader reader = new JsonReader(text)) {
+ BsonType type = reader.readBsonType();
+ if (type == BsonType.ARRAY) {
+ reader.readStartArray();
+ while ((type = reader.readBsonType()) != BsonType.END_OF_DOCUMENT) {
+ if (type != BsonType.DOCUMENT) {
+ throw new HopException(
+ BaseMessages.getString(
+ PKG,
+ "ActionMQL.NotADocument",
+ Integer.toString(commands.size() + 1),
+ type.name()));
+ }
+ commands.add(codec.decode(reader, context));
+ }
+ reader.readEndArray();
+ } else if (type == BsonType.DOCUMENT) {
+ commands.add(codec.decode(reader, context));
+ } else {
+ throw new HopException(
+ BaseMessages.getString(PKG, "ActionMQL.NotADocumentOrArray",
type.name()));
+ }
+
+ // Nothing may follow the document or the array, for example: { ... }, {
... }
+ //
+ boolean textAfter;
+ try {
+ textAfter = reader.readBsonType() != BsonType.END_OF_DOCUMENT;
+ } catch (JsonParseException e) {
+ textAfter = true;
+ }
+ if (textAfter) {
+ throw new HopException(BaseMessages.getString(PKG,
"ActionMQL.TextAfterCommands"));
+ }
+ } catch (JsonParseException e) {
+ throw new HopException(BaseMessages.getString(PKG,
"ActionMQL.InvalidJson", e.getMessage()));
+ }
+ if (commands.isEmpty()) {
+ throw new HopException(BaseMessages.getString(PKG,
"ActionMQL.NoCommandFound"));
+ }
+ return commands;
+ }
+
+ /**
+ * A command that MongoDB accepts can still fail to write documents. Those
errors are in the reply
+ * (writeErrors, writeConcernError, or nErrors for bulkWrite) while ok is 1,
so runCommand() does
+ * not throw.
+ *
+ * @param reply the reply of the command
+ * @param commandNr the number of the command (1-based)
+ * @param nrCommands the number of commands
+ * @throws MongoDbException when the reply holds write errors or a write
concern error
+ */
+ static void checkReply(Document reply, int commandNr, int nrCommands) throws
MongoDbException {
+ if (reply == null) {
+ return;
+ }
+ List<Document> writeErrors = reply.getList("writeErrors", Document.class);
+ if (writeErrors != null && !writeErrors.isEmpty()) {
+ Document first = writeErrors.get(0);
+ throw new MongoDbException(
+ BaseMessages.getString(
+ PKG,
+ "ActionMQL.WriteErrors",
+ Integer.toString(commandNr),
+ Integer.toString(nrCommands),
+ Integer.toString(writeErrors.size()),
+ String.valueOf(first.get("code")),
+ first.getString("errmsg")));
+ }
+ // The bulkWrite command counts its errors in nErrors, the errors are in
its cursor
+ //
+ if (reply.get("nErrors") instanceof Number nErrors && nErrors.longValue()
> 0) {
+ Document first = firstBulkWriteError(reply);
+ throw new MongoDbException(
+ BaseMessages.getString(
+ PKG,
+ "ActionMQL.WriteErrors",
+ Integer.toString(commandNr),
+ Integer.toString(nrCommands),
+ nErrors.toString(),
+ first == null ? "?" : String.valueOf(first.get("code")),
+ first == null ? "" : first.getString("errmsg")));
+ }
+ Document writeConcernError = reply.get("writeConcernError",
Document.class);
+ if (writeConcernError != null) {
+ throw new MongoDbException(
+ BaseMessages.getString(
+ PKG,
+ "ActionMQL.WriteConcernError",
+ Integer.toString(commandNr),
+ Integer.toString(nrCommands),
+ String.valueOf(writeConcernError.get("code")),
+ writeConcernError.getString("errmsg")));
+ }
+ }
+
+ /** The first failed operation in the first batch of a bulkWrite reply, null
if it isn't there. */
+ private static Document firstBulkWriteError(Document reply) {
+ if (!(reply.get("cursor") instanceof Document cursor)
+ || !(cursor.get("firstBatch") instanceof List<?> batch)) {
+ return null;
+ }
+ for (Object item : batch) {
+ if (item instanceof Document operation
+ && operation.get("ok") instanceof Number ok
+ && ok.doubleValue() != 1.0) {
+ return operation;
+ }
+ }
+ return null;
+ }
+
@Override
public boolean isEvaluation() {
return true;
diff --git
a/plugins/tech/mongodb/src/main/java/org/apache/hop/workflow/actions/mql/ActionMqlDialog.java
b/plugins/tech/mongodb/src/main/java/org/apache/hop/workflow/actions/mql/ActionMqlDialog.java
index caa37f74a2..132f871dac 100644
---
a/plugins/tech/mongodb/src/main/java/org/apache/hop/workflow/actions/mql/ActionMqlDialog.java
+++
b/plugins/tech/mongodb/src/main/java/org/apache/hop/workflow/actions/mql/ActionMqlDialog.java
@@ -72,6 +72,8 @@ public class ActionMqlDialog extends ActionDialog {
private Button wUseSubs;
+ private Button wFailOnWriteErrors;
+
public ActionMqlDialog(
Shell parent, ActionMql action, WorkflowMeta workflowMeta, IVariables
variables) {
super(parent, workflowMeta, variables);
@@ -255,13 +257,33 @@ public class ActionMqlDialog extends ActionDialog {
fdUseSubs.right = new FormAttachment(100, 0);
wUseSubs.setLayoutData(fdUseSubs);
+ // Fail on write errors
+ Label wlFailOnWriteErrors = new Label(shell, SWT.RIGHT);
+ wlFailOnWriteErrors.setText(BaseMessages.getString(PKG,
"ActionMQL.FailOnWriteErrors.Label"));
+ PropsUi.setLook(wlFailOnWriteErrors);
+ FormData fdlFailOnWriteErrors = new FormData();
+ fdlFailOnWriteErrors.left = new FormAttachment(0, 0);
+ fdlFailOnWriteErrors.right = new FormAttachment(middle, -margin);
+ fdlFailOnWriteErrors.top = new FormAttachment(wUseSubs, 2 * margin);
+ wlFailOnWriteErrors.setLayoutData(fdlFailOnWriteErrors);
+
+ wFailOnWriteErrors = new Button(shell, SWT.CHECK);
+ PropsUi.setLook(wFailOnWriteErrors);
+ wFailOnWriteErrors.setToolTipText(
+ BaseMessages.getString(PKG, "ActionMQL.FailOnWriteErrors.Tooltip"));
+ FormData fdFailOnWriteErrors = new FormData();
+ fdFailOnWriteErrors.left = new FormAttachment(middle, 0);
+ fdFailOnWriteErrors.top = new FormAttachment(wUseSubs, 2 * margin);
+ fdFailOnWriteErrors.right = new FormAttachment(100, 0);
+ wFailOnWriteErrors.setLayoutData(fdFailOnWriteErrors);
+
// Script line
wlMql = new Label(shell, SWT.NONE);
wlMql.setText(BaseMessages.getString(PKG, "ActionMQL.Script.Label"));
PropsUi.setLook(wlMql);
FormData fdlMql = new FormData();
fdlMql.left = new FormAttachment(0, 0);
- fdlMql.top = new FormAttachment(wUseSubs, 2 * margin);
+ fdlMql.top = new FormAttachment(wFailOnWriteErrors, 2 * margin);
wlMql.setLayoutData(fdlMql);
wMql =
@@ -321,6 +343,7 @@ public class ActionMqlDialog extends ActionDialog {
wCommandFile.setText(Const.nullToEmpty(action.getCommandFile()));
updateFileOption();
wUseSubs.setSelection(action.isUseVariableSubstitution());
+ wFailOnWriteErrors.setSelection(action.isFailOnWriteErrors());
}
private void cancel() {
@@ -345,6 +368,7 @@ public class ActionMqlDialog extends ActionDialog {
action.setUseExternalFile(wUseExternalFile.getSelection());
action.setCommandFile(wCommandFile.getText());
action.setUseVariableSubstitution(wUseSubs.getSelection());
+ action.setFailOnWriteErrors(wFailOnWriteErrors.getSelection());
action.setChanged();
diff --git
a/plugins/tech/mongodb/src/main/resources/org/apache/hop/workflow/actions/mql/messages/messages_en_US.properties
b/plugins/tech/mongodb/src/main/resources/org/apache/hop/workflow/actions/mql/messages/messages_en_US.properties
index 4340fc3b92..38bc6606d2 100644
---
a/plugins/tech/mongodb/src/main/resources/org/apache/hop/workflow/actions/mql/messages/messages_en_US.properties
+++
b/plugins/tech/mongodb/src/main/resources/org/apache/hop/workflow/actions/mql/messages/messages_en_US.properties
@@ -43,4 +43,13 @@ ActionMQL.MongoDbConnectionLabelText=MongoDB Connection
ActionMQL.MongoDbConnectionTooltipText=Select a MongoDB connection
ActionMQL.Filetype.Json=Json files
ActionMQL.Filetype.Text=Text files
-ActionMQL.Filetype.All=All files
\ No newline at end of file
+ActionMQL.Filetype.All=All files
+ActionMQL.WriteErrors=Command {0} of {1}: {2} write error(s), first error
(code {3}): {4}
+ActionMQL.WriteConcernError=Command {0} of {1}: write concern error (code
{2}): {3}
+ActionMQL.TextAfterCommands=Text after the JSON instruction is not valid: to
run several instructions, put them in one JSON array
+ActionMQL.NotADocument=Instruction {0} of the JSON array is not a JSON
document but {1}
+ActionMQL.NotADocumentOrArray=The JSON is not a document or an array of
documents but {0}
+ActionMQL.InvalidJson=The JSON is not valid: {0}
+ActionMQL.WriteErrorsIgnored={0} (ignored: Fail on write errors is not checked)
+ActionMQL.FailOnWriteErrors.Label=Fail on write errors
+ActionMQL.FailOnWriteErrors.Tooltip=Make the action false when MongoDB reports
write errors in its answer, for example an insert with a duplicate _id.\nLeave
it unchecked to accept a partial write, for example an unordered insert that
skips duplicate keys.
diff --git
a/plugins/tech/mongodb/src/test/java/org/apache/hop/workflow/actions/mql/ActionMqlTest.java
b/plugins/tech/mongodb/src/test/java/org/apache/hop/workflow/actions/mql/ActionMqlTest.java
index 45d2f065c4..5afe797835 100644
---
a/plugins/tech/mongodb/src/test/java/org/apache/hop/workflow/actions/mql/ActionMqlTest.java
+++
b/plugins/tech/mongodb/src/test/java/org/apache/hop/workflow/actions/mql/ActionMqlTest.java
@@ -20,20 +20,25 @@ package org.apache.hop.workflow.actions.mql;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
+import com.mongodb.client.MongoDatabase;
import java.util.List;
import org.apache.hop.core.Result;
+import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.metadata.SerializableMetadataProvider;
import org.apache.hop.junit.rules.RestoreHopEngineEnvironmentExtension;
import org.apache.hop.metadata.api.IHopMetadataProvider;
import org.apache.hop.metadata.api.IHopMetadataSerializer;
+import org.apache.hop.mongo.MongoDbException;
import org.apache.hop.mongo.metadata.MongoDbConnection;
import org.apache.hop.mongo.wrapper.MongoClientWrapper;
import org.bson.Document;
@@ -175,4 +180,180 @@ class ActionMqlTest {
assertEquals(1, commands.get(1).get("cmd2"));
}
}
+
+ @Test
+ void testParseCommands() throws Exception {
+ assertEquals(1, ActionMql.parseCommands(" { ping: 1 } ").size());
+ assertEquals(2, ActionMql.parseCommands("[ { \"cmd1\": 1 }, { \"cmd2\": 1
} ]").size());
+
+ // A semicolon at the end worked before and still does
+ assertEquals(1, ActionMql.parseCommands("{ ping: 1 };").size());
+ assertEquals(1, ActionMql.parseCommands("{ ping: 1 } ;\n ;\n").size());
+ assertEquals(2, ActionMql.parseCommands("[ { ping: 1 }, { ping: 2 }
];").size());
+ assertEquals(
+ "x;", ActionMql.parseCommands("{ \"insert\": \"x;\"
};").get(0).getString("insert"));
+
+ assertThrows(HopException.class, () -> ActionMql.parseCommands(null));
+ assertThrows(HopException.class, () -> ActionMql.parseCommands(" "));
+ assertThrows(HopException.class, () -> ActionMql.parseCommands(" ; "));
+ assertThrows(HopException.class, () -> ActionMql.parseCommands("[]"));
+ }
+
+ @Test
+ void testParseCommandsRejectsTextAfterTheCommand() {
+ // Text after the first document was ignored before: it is an error now,
it is never run
+ assertTextAfter("{\"drop\":\"tmp\"}, {\"drop\":\"prod\"}");
+ assertTextAfter("{ ping: 1 } { ping: 1 }");
+ assertTextAfter("{ ping: 1 } garbage");
+ assertTextAfter("[ { ping: 1 } ] garbage");
+ assertTextAfter("{ ping: 1 }; { ping: 2 }");
+ assertTextAfter("{ ping: 1 } // note");
+ assertTextAfter("{ ping: 1 }}");
+ // A repeated key can't replace the first array
+ assertTextAfter("[ {\"drop\":\"a\"} ], \"array\": [ {\"ping\":1} ]");
+ assertTextAfter("{a:1}, [{b:1}]");
+ }
+
+ private static void assertTextAfter(String mql) {
+ HopException e = assertThrows(HopException.class, () ->
ActionMql.parseCommands(mql), mql);
+ assertTrue(e.getMessage().contains("Text after the JSON instruction"),
e.getMessage());
+ }
+
+ @Test
+ void testParseCommandsRejectsValuesThatAreNotDocuments() {
+ HopException e = assertThrows(HopException.class, () ->
ActionMql.parseCommands("[1, 2]"));
+ assertTrue(e.getMessage().contains("Instruction 1 of the JSON array"),
e.getMessage());
+ e = assertThrows(HopException.class, () -> ActionMql.parseCommands("[
{a:1}, [ {b:1} ] ]"));
+ assertTrue(e.getMessage().contains("Instruction 2 of the JSON array"),
e.getMessage());
+ e = assertThrows(HopException.class, () ->
ActionMql.parseCommands("\"ping\""));
+ assertTrue(e.getMessage().contains("not a document or an array"),
e.getMessage());
+ }
+
+ @Test
+ void testParseCommandsErrorPositionMatchesTheText() {
+ // The position is the one in the text as typed, not in a wrapper around it
+ String mql = " { \"a\": 1, \"b\": # }";
+ HopException e = assertThrows(HopException.class, () ->
ActionMql.parseCommands(mql));
+ // (the parser counts from 1)
+ assertTrue(e.getMessage().contains("Position: " + (mql.indexOf('#') + 1)),
e.getMessage());
+ }
+
+ @Test
+ void testCheckReply() throws Exception {
+ ActionMql.checkReply(new Document("ok", 1.0).append("n", 2), 1, 1);
+
+ Document writeErrors =
+ new Document("ok", 1.0)
+ .append("n", 1)
+ .append(
+ "writeErrors",
+ List.of(
+ new Document("index", 1)
+ .append("code", 11000)
+ .append("errmsg", "E11000 duplicate key error")));
+ MongoDbException e =
+ assertThrows(MongoDbException.class, () ->
ActionMql.checkReply(writeErrors, 2, 3));
+ assertTrue(e.getMessage().contains("E11000"), e.getMessage());
+ assertTrue(e.getMessage().contains("2 of 3"), e.getMessage());
+
+ Document writeConcernError =
+ new Document("ok", 1.0)
+ .append("n", 1)
+ .append(
+ "writeConcernError",
+ new Document("code", 64).append("errmsg", "waiting for
replication timed out"));
+ e = assertThrows(MongoDbException.class, () ->
ActionMql.checkReply(writeConcernError, 1, 1));
+ assertTrue(e.getMessage().contains("replication"), e.getMessage());
+
+ // bulkWrite (MongoDB 8) counts its errors in nErrors, the errors are in
its cursor
+ Document bulkWrite =
+ new Document("ok", 1.0)
+ .append("nErrors", 1)
+ .append("nInserted", 1)
+ .append(
+ "cursor",
+ new Document("id", 0L)
+ .append(
+ "firstBatch",
+ List.of(
+ new Document("ok", 1.0).append("idx",
0).append("n", 1),
+ new Document("ok", 0.0)
+ .append("idx", 1)
+ .append("code", 11000)
+ .append("errmsg", "E11000 duplicate key
error"))));
+ e = assertThrows(MongoDbException.class, () ->
ActionMql.checkReply(bulkWrite, 1, 1));
+ assertTrue(e.getMessage().contains("E11000"), e.getMessage());
+ ActionMql.checkReply(new Document("ok", 1.0).append("nErrors", 0), 1, 1);
+ }
+
+ @Test
+ void testExecuteFailsOnWriteErrors() throws Exception {
+ MongoDatabase mockDb = mockDbWithWriteErrors();
+ ActionMql action = newWriteErrorsAction(mockDb, true);
+
+ Result result = action.execute(new Result(), 0);
+
+ assertFalse(result.isResult());
+ assertEquals(1, result.getNrErrors());
+ // The instruction after the failing one is not run
+ verify(mockDb, times(1)).runCommand(any(Document.class));
+ }
+
+ @Test
+ void testExecuteIgnoresWriteErrorsByDefault() throws Exception {
+ // An unordered insert that skips duplicate keys stays true, as it was
before the option
+ assertFalse(new ActionMql().isFailOnWriteErrors());
+ MongoDatabase mockDb = mockDbWithWriteErrors();
+ ActionMql action = newWriteErrorsAction(mockDb, false);
+
+ Result result = action.execute(new Result(), 0);
+
+ assertTrue(result.isResult());
+ assertEquals(0, result.getNrErrors());
+ verify(mockDb, times(2)).runCommand(any(Document.class));
+ }
+
+ private static MongoDatabase mockDbWithWriteErrors() {
+ MongoDatabase mockDb = mock(MongoDatabase.class);
+ when(mockDb.runCommand(any(Document.class)))
+ .thenReturn(
+ new Document("ok", 1.0)
+ .append("n", 1)
+ .append(
+ "writeErrors",
+ List.of(
+ new Document("index", 1)
+ .append("code", 11000)
+ .append("errmsg", "E11000 duplicate key error"))));
+ return mockDb;
+ }
+
+ private static ActionMql newWriteErrorsAction(MongoDatabase mockDb, boolean
failOnWriteErrors)
+ throws Exception {
+ ActionMql action = new ActionMql("test-write-errors");
+ IHopMetadataProvider metadataProvider = mock(IHopMetadataProvider.class);
+ IHopMetadataSerializer<MongoDbConnection> serializer =
mock(IHopMetadataSerializer.class);
+ MongoDbConnection mongoConnection = mock(MongoDbConnection.class);
+ MongoClientWrapper mockWrapper = mock(MongoClientWrapper.class);
+
+ action.setConnection("test-connection");
+ action.setMql(
+ "[ {\"insert\": \"c\", \"documents\": [{\"_id\": 1}, {\"_id\": 1}],
\"ordered\": false},"
+ + " {\"drop\": \"c\"} ]");
+ action.setFailOnWriteErrors(failOnWriteErrors);
+ action.setMetadataProvider(metadataProvider);
+
+
when(metadataProvider.getSerializer(MongoDbConnection.class)).thenReturn(serializer);
+ when(serializer.load("test-connection")).thenReturn(mongoConnection);
+ when(mongoConnection.getDbName()).thenReturn("testdb");
+ when(mongoConnection.createWrapper(any(), any())).thenReturn(mockWrapper);
+ doAnswer(
+ invocation -> {
+ org.apache.hop.mongo.wrapper.MongoDBAction<?> mongoAction =
invocation.getArgument(1);
+ return mongoAction.perform(mockDb);
+ })
+ .when(mockWrapper)
+ .perform(anyString(), any());
+ return action;
+ }
}