tmortada opened a new issue, #8138:
URL: https://github.com/apache/hop/issues/8138
### Apache Hop version?
2.19
### Java version?
21.0.12
### Operating system
Linux
### What happened?
### Title
`SparkFileOutput` with `file_format=jdbc` always sends a `path` connection
property, which some JDBC drivers (Teradata) reject
### Summary
Writing to a database through `SparkFileOutput` with `file_format=jdbc`
fails on Teradata with:
```
[Teradata JDBC Driver] [TeraJDBC 20.00.00.58] [Error 1536] [SQLState HY000]
Invalid connection parameter name path
```
The cause: `SparkFileOutputHandler` and `SparkFileIoSupport` always call
Spark's `DataFrameWriter.save(path)`, the one-argument overload that adds
`path` to the writer's options before saving. That's fine for csv/parquet/orc,
but `jdbc` is a database sink, and `path` means nothing there. Postgres and
MySQL's drivers just ignore properties they don't recognize, so this never
shows up against them. Teradata's driver validates connection properties and
rejects anything it doesn't know, so it's the one that surfaces the bug.
### Environment
- Hop 2.19.0-rc1
- Spark 4.1.3, Scala 2.13.17
- Java 21.0.12 (Alpine)
- Run config: `native-spark`, `sparkMaster=local[*]`,
`generic_transform_run_mode=DRIVER_ONLY`
- Teradata JDBC driver 20.00.00.58
- Target: Teradata Vantage Express 20.00.28.81
### How to reproduce
A minimum pipeline with two transforms attached:
1. `SparkLakeTableInput` reading any existing Iceberg table.
2. `SparkFileOutput` with:
- `file_format=jdbc`
- `file_path=/tmp/hop-bug-repro/category_sales_summary` (any non-empty
value - the transform won't run without one)
- `extra_options`:
`url=jdbc:teradata://<host>/DBS_PORT=<port>,DATABASE=<db>`, `dbtable=<table>`,
`user=<user>`, `password=<password>`, `driver=com.teradata.jdbc.TeraDriver`
Run with `hop-run.sh --runconfig native-spark --level Detailed`.
### Expected
Either the write succeeds, or it fails on something real (bad credentials,
missing table). Not a connection property Hop added on its own.
### What actually happens
```
2026/08/28 01:31:32 - bug_repro_iceberg_to_teradata_direct - ERROR: Error
executing Spark pipeline
2026/08/28 01:31:32 - bug_repro_iceberg_to_teradata_direct -
org.apache.hop.core.exception.HopException:
2026/08/28 01:31:32 - bug_repro_iceberg_to_teradata_direct - Error writing
Spark Dataset to '/tmp/hop-bug-repro/category_sales_summary' as jdbc
2026/08/28 01:31:32 - bug_repro_iceberg_to_teradata_direct - [Teradata JDBC
Driver] [TeraJDBC 20.00.00.58] [Error 1536] [SQLState HY000] Invalid connection
parameter name path
2026/08/28 01:31:32 - bug_repro_iceberg_to_teradata_direct -
at
org.apache.hop.spark.pipeline.handler.SparkFileIoSupport.writeDataset(SparkFileIoSupport.java:101)
at
org.apache.hop.spark.pipeline.handler.SparkFileOutputHandler.handleTransform(SparkFileOutputHandler.java:147)
at
org.apache.hop.spark.pipeline.HopPipelineMetaToSparkConverter.createDataset(HopPipelineMetaToSparkConverter.java:248)
at
org.apache.hop.spark.engines.SparkPipelineEngine.lambda$startThreads$0(SparkPipelineEngine.java:330)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.sql.SQLException: [Teradata JDBC Driver] [TeraJDBC
20.00.00.58] [Error 1536] [SQLState HY000] Invalid connection parameter name
path
at
com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDriverJDBCException(ErrorFactory.java:88)
at
com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDriverJDBCException(ErrorFactory.java:63)
at com.teradata.jdbc.URLParameters.setParams(URLParameters.java:745)
at com.teradata.jdbc.URLParameters.<init>(URLParameters.java:526)
at com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:203)
at com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:161)
at
org.apache.spark.sql.execution.datasources.jdbc.connection.BasicConnectionProvider.getConnection(BasicConnectionProvider.scala:50)
at
org.apache.spark.sql.execution.datasources.jdbc.connection.ConnectionProviderBase.create(ConnectionProvider.scala:102)
at
org.apache.spark.sql.jdbc.JdbcDialect.$anonfun$createConnectionFactory$1(JdbcDialects.scala:233)
at
org.apache.spark.sql.jdbc.JdbcDialect.$anonfun$createConnectionFactory$1$adapted(JdbcDialects.scala:229)
at
org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:60)
at
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:55)
...
at
org.apache.spark.sql.classic.DataFrameWriter.runCommand(DataFrameWriter.scala:592)
at
org.apache.spark.sql.classic.DataFrameWriter.save(DataFrameWriter.scala:115)
at
org.apache.hop.spark.pipeline.handler.SparkFileIoSupport.writeDataset(SparkFileIoSupport.java:98)
... 4 more
```
Full log attached as `pipeline-run.log` (also shows `SparkLakeTableInput`
reading the Iceberg table fine before this).
### Where it comes from
Checked against `apache/hop` at tag `2.19.0-rc1` ([commit
4643615](https://github.com/apache/hop/commit/46436154ae1a1e940861d485559819360c2af86e))
- the line numbers in the trace above line up exactly with this source.
[`SparkFileIoSupport.writeDataset`](https://github.com/apache/hop/blob/46436154ae1a1e940861d485559819360c2af86e/plugins/engines/spark/src/main/java/org/apache/hop/spark/pipeline/handler/SparkFileIoSupport.java#L77-L105):
```java
public static void writeDataset(
Dataset<Row> dataset,
String format,
String path,
SaveMode saveMode,
Map<String, String> options,
String[] partitionColumns,
Integer coalesce)
throws HopException {
Dataset<Row> toWrite = dataset;
if (coalesce != null && coalesce > 0) {
toWrite = toWrite.coalesce(coalesce);
}
DataFrameWriter<Row> writer =
toWrite.write().mode(saveMode).format(format);
for (Map.Entry<String, String> e : options.entrySet()) {
writer = writer.option(e.getKey(), e.getValue());
}
if (partitionColumns != null && partitionColumns.length > 0) {
writer = writer.partitionBy(partitionColumns);
}
try {
writer.save(path); // line 98
} catch (Exception e) {
throw new HopException(
SparkPathDialect.withPathHint(
"Error writing Spark Dataset to '" + path + "' as " + format,
path),
e);
}
}
```
`format` and `path` are just passed straight through, no branching on what
`format` is. `save(path)` on line 98 is Spark's own path-argument overload,
which sets `path` as an option before saving - that's the one that reaches
Teradata's driver as an unrecognized connection parameter.
There's a second, earlier problem too. Take `file_path` out of the transform
entirely and it doesn't skip the bug, it just fails sooner, somewhere else:
```
org.apache.hop.core.exception.HopException:
Spark File Output 'Write JDBC Direct to Teradata' has no file path configured
at
org.apache.hop.spark.pipeline.handler.SparkFileOutputHandler.handleTransform(SparkFileOutputHandler.java:95)
at
org.apache.hop.spark.pipeline.HopPipelineMetaToSparkConverter.createDataset(HopPipelineMetaToSparkConverter.java:248)
at
org.apache.hop.spark.engines.SparkPipelineEngine.lambda$startThreads$0(SparkPipelineEngine.java:330)
at java.base/java.lang.Thread.run(Thread.java:1583)
```
(log: `pipeline-run-no-filepath.log`, pipeline:
`repro-pipeline-no-filepath.hpl`. Same result with `<file_path/>` left empty
instead of removed - `pipeline-run-empty-filepath.log`.)
That's [`SparkFileOutputHandler`, lines
91-95](https://github.com/apache/hop/blob/46436154ae1a1e940861d485559819360c2af86e/plugins/engines/spark/src/main/java/org/apache/hop/spark/pipeline/handler/SparkFileOutputHandler.java#L91-L95):
```java
String resolved = variables.resolve(meta.getFilePath());
String path = SparkPathDialect.toSparkUri(resolved, runConfiguration);
if (StringUtils.isEmpty(path)) { // line 93
throw new HopException(
"Spark File Output '" + transformMeta.getName() + "' has no file path
configured");
}
```
So there are really two separate issues combined: `file_path` is required no
matter what format you pick (line 93), and then whatever value you're forced to
give it gets sent to the driver as a `path` connection property (line 98).
Fixing one without the other doesn't help - drop the line 93 check and you just
hit line 98 instead.
### Attached
- `repro-pipeline.hpl`, `pipeline-run.log` - the main repro of the issue
- `repro-pipeline-no-filepath.hpl`, `pipeline-run-no-filepath.log` -
file_path removed
- `repro-pipeline-empty-filepath.hpl`, `pipeline-run-empty-filepath.log` -
file_path left empty
### Environment
* Java: OpenJDK 64-Bit Server VM (build 21.0.12+8-alpine-r0, mixed mode,
sharing)
* scala-library-2.13.17.jar
* spark-core_2.13-4.1.3.jar
* -rw-rw-r-- 1 1000 1000 1294820 Aug 20 20:18 /opt/hop/lib/jdbc/terajdbc4.jar
* -rw-rw-r-- 1 1000 1000 1294820 Aug 20 20:18
/opt/hop/plugins/engines/spark/lib/terajdbc4.jar
* Teradata JDBC driver Implementation-Version: 20.00.00.58
[pipeline-run.zip](https://github.com/user-attachments/files/31538393/pipeline-run.zip)
### Issue Priority
Priority: 2
### Issue Component
Component: Transforms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
