This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new a993ce6397 Bulk load from MySQL into file: fix High Priority export
and unknown … (#9354)
a993ce6397 is described below
commit a993ce63979db7ecfd489c722e8f476acbaed2fa
Author: Bart Maertens <[email protected]>
AuthorDate: Thu Nov 12 09:50:07 2026 +0100
Bulk load from MySQL into file: fix High Priority export and unknown …
(#9354)
* Bulk load from MySQL into file: fix High Priority export and unknown
connection NPE, fixes #9166
* Address review on #9354: drop LOCK IN SHARE MODE, escape the file name,
guard unknown and empty connections in both MySQL bulk actions
---
.../integration-tests-database.yaml | 5 +
.../ROOT/pages/workflow/actions/mysqlbulkfile.adoc | 8 +-
.../database/main-0050-mysql-bulk-file.hwf | 276 +++++++++++++++++++++
.../actions/mysqlbulkfile/ActionMysqlBulkFile.java | 37 ++-
.../actions/mysqlbulkload/ActionMysqlBulkLoad.java | 23 +-
.../messages/messages_en_US.properties | 1 +
.../messages/messages_en_US.properties | 1 +
.../ActionMysqlBulkFileExecuteTest.java | 191 ++++++++++++++
.../ActionMysqlBulkLoadUnknownConnectionTest.java | 95 +++++++
9 files changed, 610 insertions(+), 27 deletions(-)
diff --git a/docker/integration-tests/integration-tests-database.yaml
b/docker/integration-tests/integration-tests-database.yaml
index deb2593d6e..1f8c524978 100644
--- a/docker/integration-tests/integration-tests-database.yaml
+++ b/docker/integration-tests/integration-tests-database.yaml
@@ -65,3 +65,8 @@ services:
- MYSQL_ROOT_PASSWORD=my-secret-pw
volumes:
- ./resource/mysql/my.cnf:/etc/mysql/conf.d/my.cnf
+ tmpfs:
+ # secure_file_priv folder, where main-0050 lets the server write its
SELECT ... INTO OUTFILE
+ # files. INTO OUTFILE refuses to overwrite a file, so it has to be empty
on every start, also
+ # when KEEP_IMAGES=true makes compose reuse the container.
+ - /var/lib/mysql-files:mode=1777
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/mysqlbulkfile.adoc
b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/mysqlbulkfile.adoc
index c8a0568202..7caf252bbc 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/mysqlbulkfile.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/mysqlbulkfile.adoc
@@ -34,7 +34,7 @@ The `Bulk load from MySQL into file` action bulk loads data
from a MySQL table i
|Source schema|The schema of the table to export.
|Source table|The table to export.
|Target File name|The file to write, on the database server.
-|High Priority|Adds `HIGH_PRIORITY` to the query. Checked by default: see the
NOTE under <<Result>>.
+|High Priority|Adds `HIGH_PRIORITY` to the query (checked by default). This
only has an effect on storage engines that use table-level locking, such as
MyISAM.
|Type|OUTFILE (default) writes the rows with the separators below. DUMPFILE
writes a single row without any formatting: more than one row makes the action
fail.
|Field separator|The field separator to use.
|Fields enclosed by|The field enclosure to use.
@@ -58,15 +58,11 @@ What this action passes on to the next action. See
xref:workflow/action-results.
|===
|Result|Value
|True|The file was written, with 0 errors. Also true when the file already
exists and *If file exists* is *Do Nothing*: then nothing is exported, and the
action doesn't connect to the database.
-|False|The file exists and *If file exists* is *Fail*, the table doesn't
exist, *Target File name* is empty, the connection can't be opened, or MySQL
refuses the query (for example a path outside the folder allowed by the
server's `secure_file_priv` setting, or *DUMPFILE* on more than one row). The
number of errors is 1.
+|False|The file exists and *If file exists* is *Fail*, the table doesn't
exist, *Target File name* is empty, the connection doesn't exist or can't be
opened, or MySQL refuses the query (for example a path outside the folder
allowed by the server's `secure_file_priv` setting, or *DUMPFILE* on more than
one row). The number of errors is 1.
|Result files|The target file is added when *Add file to result filenames* is
checked and the file was written. Otherwise passed on unchanged.
|Result rows|Passed on unchanged.
|===
-NOTE: *High Priority* is checked by default, and on MySQL 8.4 it makes the
action fail with "Table ... appears in multiple locking clauses": the action
always adds `LOCK IN SHARE MODE` to the query, and MySQL doesn't accept both.
Uncheck *High Priority* to export.
-
-NOTE: A *Connection* that doesn't exist in the project is not a false result:
the workflow stops with an error and no hop from the action is followed.
-
NOTE: When the table doesn't exist, the reason is only logged at the Detailed
log level: at the Basic level, the log shows a false action without an error
message.
NOTE: The MySQL server writes the file (`SELECT ... INTO OUTFILE`), so *Target
File name* is a path on the database server. *If file exists* is checked on the
machine where Hop runs.
diff --git a/integration-tests/database/main-0050-mysql-bulk-file.hwf
b/integration-tests/database/main-0050-mysql-bulk-file.hwf
new file mode 100644
index 0000000000..208b616e5d
--- /dev/null
+++ b/integration-tests/database/main-0050-mysql-bulk-file.hwf
@@ -0,0 +1,276 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<workflow>
+ <name>main-0050-mysql-bulk-file</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>Bulk load from MySQL into file (issue #9166): export with and
without High Priority, and a connection that does not exist is a false result,
not a stopped workflow.</description>
+ <extended_description>The MySQL server writes the files, into its
secure_file_priv folder /var/lib/mysql-files (a tmpfs in the compose file, so
it is empty on every start). LOAD_FILE() reads them back on the
server.</extended_description>
+ <workflow_version/>
+ <created_user>-</created_user>
+ <created_date>2026/11/05 12:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/11/05 12:00:00.000</modified_date>
+ <parameters>
+ </parameters>
+ <actions>
+ <action>
+ <name>Start</name>
+ <description/>
+ <type>SPECIAL</type>
+ <attributes/>
+ <DayOfMonth>1</DayOfMonth>
+ <doNotWaitOnFirstExecution>N</doNotWaitOnFirstExecution>
+ <hour>12</hour>
+ <intervalMinutes>60</intervalMinutes>
+ <intervalSeconds>0</intervalSeconds>
+ <minutes>0</minutes>
+ <repeat>N</repeat>
+ <schedulerType>0</schedulerType>
+ <weekDay>1</weekDay>
+ <parallel>N</parallel>
+ <xloc>80</xloc>
+ <yloc>80</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Init table</name>
+ <description/>
+ <type>SQL</type>
+ <attributes/>
+ <connection>mysql</connection>
+ <sendOneStatement>N</sendOneStatement>
+ <sql>DROP TABLE IF EXISTS bulk_file_source;
+
+CREATE TABLE bulk_file_source
+(
+ id INT,
+ name VARCHAR(20)
+) ENGINE = InnoDB;
+
+INSERT INTO bulk_file_source (id, name)
+VALUES (1, 'Goku'),
+ (2, 'Vegeta');
+</sql>
+ <sqlfromfile>N</sqlfromfile>
+ <useVariableSubstitution>N</useVariableSubstitution>
+ <parallel>N</parallel>
+ <xloc>240</xloc>
+ <yloc>80</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Export with High Priority</name>
+ <description/>
+ <type>MYSQL_BULK_FILE</type>
+ <attributes/>
+ <addfiletoresult>N</addfiletoresult>
+ <connection>mysql</connection>
+ <enclosed/>
+ <filename>/var/lib/mysql-files/0050-high-priority.txt</filename>
+ <highpriority>Y</highpriority>
+ <iffileexists>2</iffileexists>
+ <limitlines>0</limitlines>
+ <lineterminated/>
+ <listcolumn/>
+ <optionenclosed>N</optionenclosed>
+ <outdumpvalue>0</outdumpvalue>
+ <schemaname/>
+ <separator>;</separator>
+ <tablename>bulk_file_source</tablename>
+ <parallel>N</parallel>
+ <xloc>416</xloc>
+ <yloc>80</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Check High Priority file</name>
+ <description/>
+ <type>EVAL_TABLE_CONTENT</type>
+ <attributes/>
+ <connection>mysql</connection>
+ <schemaname/>
+ <tablename/>
+ <success_condition>rows_count_equal</success_condition>
+ <limit>1</limit>
+ <is_custom_sql>Y</is_custom_sql>
+ <is_usevars>N</is_usevars>
+ <custom_sql>SELECT 1 FROM DUAL
+WHERE LOAD_FILE('/var/lib/mysql-files/0050-high-priority.txt') =
'1;Goku\n2;Vegeta\n'</custom_sql>
+ <add_rows_result>N</add_rows_result>
+ <clear_result_rows>Y</clear_result_rows>
+ <parallel>N</parallel>
+ <xloc>608</xloc>
+ <yloc>80</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Export without High Priority</name>
+ <description/>
+ <type>MYSQL_BULK_FILE</type>
+ <attributes/>
+ <addfiletoresult>N</addfiletoresult>
+ <connection>mysql</connection>
+ <enclosed/>
+
<filename>/var/lib/mysql-files/0050-normal-priority-o'brien.txt</filename>
+ <highpriority>N</highpriority>
+ <iffileexists>2</iffileexists>
+ <limitlines>0</limitlines>
+ <lineterminated/>
+ <listcolumn/>
+ <optionenclosed>N</optionenclosed>
+ <outdumpvalue>0</outdumpvalue>
+ <schemaname/>
+ <separator>;</separator>
+ <tablename>bulk_file_source</tablename>
+ <parallel>N</parallel>
+ <xloc>416</xloc>
+ <yloc>208</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Check file without High Priority</name>
+ <description/>
+ <type>EVAL_TABLE_CONTENT</type>
+ <attributes/>
+ <connection>mysql</connection>
+ <schemaname/>
+ <tablename/>
+ <success_condition>rows_count_equal</success_condition>
+ <limit>1</limit>
+ <is_custom_sql>Y</is_custom_sql>
+ <is_usevars>N</is_usevars>
+ <custom_sql>SELECT 1 FROM DUAL
+WHERE LOAD_FILE('/var/lib/mysql-files/0050-normal-priority-o''brien.txt') =
'1;Goku\n2;Vegeta\n'</custom_sql>
+ <add_rows_result>N</add_rows_result>
+ <clear_result_rows>Y</clear_result_rows>
+ <parallel>N</parallel>
+ <xloc>608</xloc>
+ <yloc>208</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Export with unknown connection</name>
+ <description/>
+ <type>MYSQL_BULK_FILE</type>
+ <attributes/>
+ <addfiletoresult>N</addfiletoresult>
+ <connection>does-not-exist</connection>
+ <enclosed/>
+ <filename>/var/lib/mysql-files/0050-unknown-connection.txt</filename>
+ <highpriority>N</highpriority>
+ <iffileexists>2</iffileexists>
+ <limitlines>0</limitlines>
+ <lineterminated/>
+ <listcolumn/>
+ <optionenclosed>N</optionenclosed>
+ <outdumpvalue>0</outdumpvalue>
+ <schemaname/>
+ <separator>;</separator>
+ <tablename>bulk_file_source</tablename>
+ <parallel>N</parallel>
+ <xloc>416</xloc>
+ <yloc>336</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Success</name>
+ <description/>
+ <type>SUCCESS</type>
+ <attributes/>
+ <parallel>N</parallel>
+ <xloc>608</xloc>
+ <yloc>336</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Abort: unknown connection was true</name>
+ <description/>
+ <type>ABORT</type>
+ <attributes/>
+ <always_log_rows>Y</always_log_rows>
+ <message>The export with a connection that does not exist gave a true
result</message>
+ <parallel>N</parallel>
+ <xloc>416</xloc>
+ <yloc>464</yloc>
+ <attributes_hac/>
+ </action>
+ </actions>
+ <hops>
+ <hop>
+ <from>Start</from>
+ <to>Init table</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>Y</unconditional>
+ </hop>
+ <hop>
+ <from>Init table</from>
+ <to>Export with High Priority</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>Export with High Priority</from>
+ <to>Check High Priority file</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>Check High Priority file</from>
+ <to>Export without High Priority</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>Export without High Priority</from>
+ <to>Check file without High Priority</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>Check file without High Priority</from>
+ <to>Export with unknown connection</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>Export with unknown connection</from>
+ <to>Success</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>Export with unknown connection</from>
+ <to>Abort: unknown connection was true</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ </hops>
+ <notepads>
+ </notepads>
+ <attributes/>
+</workflow>
diff --git
a/plugins/databases/mysql/src/main/java/org/apache/hop/workflow/actions/mysqlbulkfile/ActionMysqlBulkFile.java
b/plugins/databases/mysql/src/main/java/org/apache/hop/workflow/actions/mysqlbulkfile/ActionMysqlBulkFile.java
index d10321836f..05c7411412 100644
---
a/plugins/databases/mysql/src/main/java/org/apache/hop/workflow/actions/mysqlbulkfile/ActionMysqlBulkFile.java
+++
b/plugins/databases/mysql/src/main/java/org/apache/hop/workflow/actions/mysqlbulkfile/ActionMysqlBulkFile.java
@@ -17,7 +17,6 @@
package org.apache.hop.workflow.actions.mysqlbulkfile;
-import java.io.File;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.List;
@@ -163,9 +162,18 @@ public class ActionMysqlBulkFile extends ActionBase {
if (fileName != null) {
// User has specified a file, We can continue ...
String realFilename = getRealFilename();
- File file = new File(realFilename);
+ boolean fileExists;
+ try {
+ // An empty name would resolve to the working folder, which always
exists
+ fileExists =
+ !Utils.isEmpty(realFilename) && HopVfs.fileExists(realFilename,
getVariables());
+ } catch (HopFileException e) {
+ result.setNrErrors(1);
+ logError(BaseMessages.getString(PKG,
CONST_ACTION_MYSQL_BULK_FILE_ERROR_LABEL), e);
+ return result;
+ }
- if (file.exists() && ifFileExists == 2) {
+ if (fileExists && ifFileExists == 2) {
// the file exists and user want to Fail
result.setResult(false);
result.setNrErrors(1);
@@ -174,7 +182,7 @@ public class ActionMysqlBulkFile extends ActionBase {
+ realFilename
+ BaseMessages.getString(PKG,
CONST_ACTION_MYSQL_BULK_FILE_FILE_EXISTS_2_LABEL));
- } else if (file.exists() && ifFileExists == 1) {
+ } else if (fileExists && ifFileExists == 1) {
// the file exists and user want to do nothing
result.setResult(true);
if (isDetailed()) {
@@ -186,7 +194,7 @@ public class ActionMysqlBulkFile extends ActionBase {
} else {
- if (file.exists() && ifFileExists == 0) {
+ if (fileExists && ifFileExists == 0) {
// File exists and user want to renamme it with unique name
// Format Date
@@ -221,8 +229,15 @@ public class ActionMysqlBulkFile extends ActionBase {
+ BaseMessages.getString(PKG,
CONST_ACTION_MYSQL_BULK_FILE_FILE_EXISTS_2_LABEL));
}
- if (connection != null) {
+ if (!Utils.isEmpty(connection)) {
DatabaseMeta databaseMeta =
parentWorkflowMeta.findDatabase(connection, getVariables());
+ if (databaseMeta == null) {
+ result.setNrErrors(1);
+ logError(
+ BaseMessages.getString(
+ PKG, "ActionMysqlBulkFile.Error.ConnectionNotFound",
connection));
+ return result;
+ }
// User has specified a connection, We can continue ...
try (Database db = new Database(this, this, databaseMeta)) {
@@ -301,7 +316,7 @@ public class ActionMysqlBulkFile extends ActionBase {
+ " "
+ outDumpText
+ " '"
- + realFilename
+ + Const.replace(realFilename, "'", "''")
+ "' "
+ fieldSeparator
+ " "
@@ -311,8 +326,7 @@ public class ActionMysqlBulkFile extends ActionBase {
+ " FROM "
+ realTablename
+ " "
- + limitNbrLignes
- + " LOCK IN SHARE MODE";
+ + limitNbrLignes;
try {
if (isDetailed()) {
@@ -432,8 +446,9 @@ public class ActionMysqlBulkFile extends ActionBase {
public List<ResourceReference> getResourceDependencies(
IVariables variables, WorkflowMeta workflowMeta) {
List<ResourceReference> references =
super.getResourceDependencies(variables, workflowMeta);
- if (connection != null) {
- DatabaseMeta databaseMeta = parentWorkflowMeta.findDatabase(connection,
getVariables());
+ DatabaseMeta databaseMeta =
+ Utils.isEmpty(connection) ? null :
workflowMeta.findDatabase(connection, variables);
+ if (databaseMeta != null) {
ResourceReference reference = new ResourceReference(this);
reference
.getEntries()
diff --git
a/plugins/databases/mysql/src/main/java/org/apache/hop/workflow/actions/mysqlbulkload/ActionMysqlBulkLoad.java
b/plugins/databases/mysql/src/main/java/org/apache/hop/workflow/actions/mysqlbulkload/ActionMysqlBulkLoad.java
index beb546abe1..f5efec6798 100644
---
a/plugins/databases/mysql/src/main/java/org/apache/hop/workflow/actions/mysqlbulkload/ActionMysqlBulkLoad.java
+++
b/plugins/databases/mysql/src/main/java/org/apache/hop/workflow/actions/mysqlbulkload/ActionMysqlBulkLoad.java
@@ -197,8 +197,16 @@ public class ActionMysqlBulkLoad extends ActionBase {
logDetailed("File [" + realFilename + "] exists.");
}
- if (connection != null) {
- DatabaseMeta databaseMeta =
parentWorkflowMeta.findDatabase(connection, getVariables());
+ DatabaseMeta databaseMeta =
+ Utils.isEmpty(connection)
+ ? null
+ : parentWorkflowMeta.findDatabase(connection,
getVariables());
+ if (!Utils.isEmpty(connection) && databaseMeta == null) {
+ result.setNrErrors(1);
+ logError(
+ BaseMessages.getString(
+ PKG, "ActionMysqlBulkLoad.Error.ConnectionNotFound",
connection));
+ } else if (databaseMeta != null) {
// User has specified a connection, We can continue ...
try (Database db = new Database(this, this, databaseMeta)) {
db.connect();
@@ -428,19 +436,14 @@ public class ActionMysqlBulkLoad extends ActionBase {
return returnString;
}
- @SuppressWarnings("javabugs:S2259") // the reference is only built for a
resolved connection
@Override
public List<ResourceReference> getResourceDependencies(
IVariables variables, WorkflowMeta workflowMeta) {
List<ResourceReference> references =
super.getResourceDependencies(variables, workflowMeta);
ResourceReference reference = null;
- if (connection != null) {
- DatabaseMeta databaseMeta = null;
- try {
- databaseMeta = DatabaseMeta.loadDatabase(getMetadataProvider(),
connection);
- } catch (Exception e) {
- logError("Unable to load database :" + connection, e);
- }
+ DatabaseMeta databaseMeta =
+ Utils.isEmpty(connection) ? null :
workflowMeta.findDatabase(connection, variables);
+ if (databaseMeta != null) {
reference = new ResourceReference(this);
references.add(reference);
reference
diff --git
a/plugins/databases/mysql/src/main/resources/org/apache/hop/workflow/actions/mysqlbulkfile/messages/messages_en_US.properties
b/plugins/databases/mysql/src/main/resources/org/apache/hop/workflow/actions/mysqlbulkfile/messages/messages_en_US.properties
index ddb13aa322..a7d4debf19 100644
---
a/plugins/databases/mysql/src/main/resources/org/apache/hop/workflow/actions/mysqlbulkfile/messages/messages_en_US.properties
+++
b/plugins/databases/mysql/src/main/resources/org/apache/hop/workflow/actions/mysqlbulkfile/messages/messages_en_US.properties
@@ -24,6 +24,7 @@ ActionMysqlBulkFile.Description=Bulk export a MySQL table
into a file
ActionMysqlBulkFile.Do_Nothing_IfFileExists.Label=Do Nothing
ActionMysqlBulkFile.DumpFileValue.Label=DUMPFILE
ActionMysqlBulkFile.Enclosed.Label=Fields enclosed by
+ActionMysqlBulkFile.Error.ConnectionNotFound=Database connection [{0}] could
not be found.
ActionMysqlBulkFile.Error.Label=An error occurred executing this action :
ActionMysqlBulkFile.Fail_IfFileExists.Label=Fail
ActionMysqlBulkFile.FileExists1.Label=The target file [
diff --git
a/plugins/databases/mysql/src/main/resources/org/apache/hop/workflow/actions/mysqlbulkload/messages/messages_en_US.properties
b/plugins/databases/mysql/src/main/resources/org/apache/hop/workflow/actions/mysqlbulkload/messages/messages_en_US.properties
index 1919810e93..3310bbfc1e 100644
---
a/plugins/databases/mysql/src/main/resources/org/apache/hop/workflow/actions/mysqlbulkload/messages/messages_en_US.properties
+++
b/plugins/databases/mysql/src/main/resources/org/apache/hop/workflow/actions/mysqlbulkload/messages/messages_en_US.properties
@@ -21,6 +21,7 @@ ActionMysqlBulkLoad.ConnectionError2.DialogMessage=Please
select a valid databas
ActionMysqlBulkLoad.ConProrityValue.Label=CONCURRENT
ActionMysqlBulkLoad.Description=Load data from a file into a Mysql table
ActionMysqlBulkLoad.Enclosed.Label=Fields enclosed by
+ActionMysqlBulkLoad.Error.ConnectionNotFound=Database connection [{0}] could
not be found.
ActionMysqlBulkLoad.Escaped.Label=Fields escaped by
ActionMysqlBulkLoad.Escaped.Tooltip=If the fields escaped by is not empty,
occurrences of that character are stripped\n and the following character is
taken literally as part of a field value.
ActionMysqlBulkLoad.Filename.Label=Source File name
diff --git
a/plugins/databases/mysql/src/test/java/org/apache/hop/workflow/actions/mysqlbulkfile/ActionMysqlBulkFileExecuteTest.java
b/plugins/databases/mysql/src/test/java/org/apache/hop/workflow/actions/mysqlbulkfile/ActionMysqlBulkFileExecuteTest.java
new file mode 100644
index 0000000000..a753ad026b
--- /dev/null
+++
b/plugins/databases/mysql/src/test/java/org/apache/hop/workflow/actions/mysqlbulkfile/ActionMysqlBulkFileExecuteTest.java
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.workflow.actions.mysqlbulkfile;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.mockConstruction;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.sql.PreparedStatement;
+import java.util.List;
+import org.apache.hop.core.HopEnvironment;
+import org.apache.hop.core.Result;
+import org.apache.hop.core.database.Database;
+import org.apache.hop.core.database.DatabaseMeta;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.variables.Variables;
+import org.apache.hop.junit.rules.RestoreHopEngineEnvironmentExtension;
+import org.apache.hop.resource.ResourceReference;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+import org.mockito.ArgumentCaptor;
+import org.mockito.MockedConstruction;
+
+/** Issue #9166: High Priority export statement and a connection that does not
exist. */
+class ActionMysqlBulkFileExecuteTest {
+ @RegisterExtension
+ static RestoreHopEngineEnvironmentExtension env = new
RestoreHopEngineEnvironmentExtension();
+
+ @TempDir Path tempDir;
+
+ @BeforeAll
+ static void setUpBeforeClass() throws HopException {
+ HopEnvironment.init();
+ }
+
+ /** MySQL 8+ rejects HIGH_PRIORITY together with a locking clause (error
3569). */
+ @Test
+ void highPriorityHasNoLockingClause() throws Exception {
+ String sql = runAndCaptureSql(newAction(true));
+
+ assertTrue(sql.contains("HIGH_PRIORITY"), sql);
+ assertFalse(sql.contains("LOCK IN SHARE MODE"), sql);
+ }
+
+ @Test
+ void withoutHighPriorityThereIsNoLockingClauseEither() throws Exception {
+ String sql = runAndCaptureSql(newAction(false));
+
+ assertFalse(sql.contains("HIGH_PRIORITY"), sql);
+ assertFalse(sql.contains("LOCK IN SHARE MODE"), sql);
+ }
+
+ @Test
+ void quoteInTheFileNameIsEscaped() throws Exception {
+ ActionMysqlBulkFile action = newAction(false);
+
action.setFileName(tempDir.resolve("o'brien.txt").toAbsolutePath().toString());
+
+ String sql = runAndCaptureSql(action);
+
+ assertTrue(sql.contains("o''brien.txt' "), sql);
+ }
+
+ @Test
+ void existingFileWithFailIsAFalseResultWithoutConnecting() throws Exception {
+ ActionMysqlBulkFile action = newAction(false);
+ Files.writeString(tempDir.resolve("out.txt"), "x");
+ action.ifFileExists = 2;
+
+ try (MockedConstruction<Database> databases =
mockConstruction(Database.class)) {
+ Result result = action.execute(new Result(), 0);
+
+ assertFalse(result.getResult());
+ assertEquals(1, result.getNrErrors());
+ assertTrue(databases.constructed().isEmpty());
+ }
+ }
+
+ @Test
+ void existingFileWithDoNothingIsATrueResultWithoutConnecting() throws
Exception {
+ ActionMysqlBulkFile action = newAction(false);
+ Files.writeString(tempDir.resolve("out.txt"), "x");
+ action.ifFileExists = 1;
+
+ try (MockedConstruction<Database> databases =
mockConstruction(Database.class)) {
+ Result result = action.execute(new Result(), 0);
+
+ assertTrue(result.getResult());
+ assertEquals(0, result.getNrErrors());
+ assertTrue(databases.constructed().isEmpty());
+ }
+ }
+
+ /** Not mocking Database: its constructor threw the NullPointerException
that stopped the run. */
+ @Test
+ void unknownConnectionIsAFalseResultWithOneError() {
+ ActionMysqlBulkFile action = newAction(false);
+ action.setConnection("does-not-exist");
+ WorkflowMeta workflowMeta = mock(WorkflowMeta.class);
+ action.setParentWorkflowMeta(workflowMeta);
+
+ Result result = action.execute(new Result(), 0);
+
+ assertFalse(result.getResult());
+ assertEquals(1, result.getNrErrors());
+ }
+
+ @Test
+ void emptyConnectionIsNotLookedUp() {
+ ActionMysqlBulkFile action = newAction(false);
+ action.setConnection("");
+ WorkflowMeta workflowMeta = mock(WorkflowMeta.class);
+ action.setParentWorkflowMeta(workflowMeta);
+
+ Result result = action.execute(new Result(), 0);
+
+ assertFalse(result.getResult());
+ assertEquals(1, result.getNrErrors());
+ verify(workflowMeta, never()).findDatabase(anyString(), any());
+ }
+
+ @Test
+ void unknownConnectionHasNoResourceDependency() {
+ ActionMysqlBulkFile action = newAction(false);
+ action.setConnection("does-not-exist");
+ WorkflowMeta workflowMeta = mock(WorkflowMeta.class);
+ action.setParentWorkflowMeta(workflowMeta);
+
+ List<ResourceReference> references =
+ action.getResourceDependencies(new Variables(), workflowMeta);
+
+ assertTrue(references.isEmpty());
+ }
+
+ private ActionMysqlBulkFile newAction(boolean highPriority) {
+ ActionMysqlBulkFile action = new ActionMysqlBulkFile("bulkfile");
+ action.setTableName("mytable");
+ action.setFileName(tempDir.resolve("out.txt").toAbsolutePath().toString());
+ action.setConnection("mysql");
+ action.setHighPriority(highPriority);
+ return action;
+ }
+
+ private String runAndCaptureSql(ActionMysqlBulkFile action) throws Exception
{
+ WorkflowMeta workflowMeta = mock(WorkflowMeta.class);
+ when(workflowMeta.findDatabase(eq("mysql"),
any())).thenReturn(mock(DatabaseMeta.class));
+ action.setParentWorkflowMeta(workflowMeta);
+
+ try (MockedConstruction<Database> databases =
+ mockConstruction(
+ Database.class,
+ (db, context) -> {
+ when(db.checkTableExists(any(), any())).thenReturn(true);
+
when(db.prepareSql(anyString())).thenReturn(mock(PreparedStatement.class));
+ })) {
+ Result result = action.execute(new Result(), 0);
+
+ assertTrue(result.getResult());
+ ArgumentCaptor<String> sql = ArgumentCaptor.forClass(String.class);
+ verify(databases.constructed().get(0)).prepareSql(sql.capture());
+ return sql.getValue();
+ }
+ }
+}
diff --git
a/plugins/databases/mysql/src/test/java/org/apache/hop/workflow/actions/mysqlbulkload/ActionMysqlBulkLoadUnknownConnectionTest.java
b/plugins/databases/mysql/src/test/java/org/apache/hop/workflow/actions/mysqlbulkload/ActionMysqlBulkLoadUnknownConnectionTest.java
new file mode 100644
index 0000000000..fe580393f6
--- /dev/null
+++
b/plugins/databases/mysql/src/test/java/org/apache/hop/workflow/actions/mysqlbulkload/ActionMysqlBulkLoadUnknownConnectionTest.java
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.workflow.actions.mysqlbulkload;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.mockConstruction;
+
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import org.apache.hop.core.HopEnvironment;
+import org.apache.hop.core.Result;
+import org.apache.hop.core.database.Database;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.variables.Variables;
+import org.apache.hop.junit.rules.RestoreHopEngineEnvironmentExtension;
+import org.apache.hop.resource.ResourceEntry;
+import org.apache.hop.resource.ResourceEntry.ResourceType;
+import org.apache.hop.resource.ResourceReference;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+import org.junit.jupiter.api.io.TempDir;
+import org.mockito.MockedConstruction;
+
+/** A connection that does not exist, the same case as #9166 in Bulk load from
MySQL into file. */
+class ActionMysqlBulkLoadUnknownConnectionTest {
+ @RegisterExtension
+ static RestoreHopEngineEnvironmentExtension env = new
RestoreHopEngineEnvironmentExtension();
+
+ @TempDir Path tempDir;
+
+ @BeforeAll
+ static void setUpBeforeClass() throws HopException {
+ HopEnvironment.init();
+ }
+
+ @Test
+ void unknownConnectionIsAFalseResultWithoutOpeningADatabase() throws
Exception {
+ ActionMysqlBulkLoad action = newAction();
+ action.setParentWorkflowMeta(mock(WorkflowMeta.class));
+
+ try (MockedConstruction<Database> databases =
mockConstruction(Database.class)) {
+ Result result = action.execute(new Result(), 0);
+
+ assertFalse(result.getResult());
+ assertEquals(1, result.getNrErrors());
+ assertTrue(databases.constructed().isEmpty());
+ }
+ }
+
+ @Test
+ void unknownConnectionOnlyHasTheFileAsResourceDependency() throws Exception {
+ ActionMysqlBulkLoad action = newAction();
+ WorkflowMeta workflowMeta = mock(WorkflowMeta.class);
+ action.setParentWorkflowMeta(workflowMeta);
+
+ List<ResourceReference> references =
+ action.getResourceDependencies(new Variables(), workflowMeta);
+
+ assertEquals(1, references.size());
+ List<ResourceEntry> entries = references.get(0).getEntries();
+ assertEquals(1, entries.size());
+ assertEquals(ResourceType.FILE, entries.get(0).getResourcetype());
+ }
+
+ private ActionMysqlBulkLoad newAction() throws Exception {
+ Path dataFile = Files.writeString(tempDir.resolve("data.csv"), "1;2\n");
+ ActionMysqlBulkLoad action = new ActionMysqlBulkLoad("bulkload");
+ action.setFileName(dataFile.toAbsolutePath().toString());
+ action.setTableName("mytable");
+ action.setConnection("does-not-exist");
+ action.setLocalInFile(true);
+ return action;
+ }
+}