vbhanuchander-lang opened a new pull request, #8176:
URL: https://github.com/apache/hop/pull/8176
Fixes #8138
### The defect
`SparkFileIoSupport.writeDataset` always called the one-argument
`DataFrameWriter.save(path)`:
```java
writer.save(path); // SparkFileIoSupport.java:98
```
That overload stores its argument as the `path` writer option before saving.
Spark's JDBC provider forwards every option it does not recognise to the driver
as a **connection property**, so a JDBC sink — which has no path — receives one
anyway.
Postgres and MySQL ignore connection properties they do not know, so this
has been invisible against them. Teradata validates them and fails the write:
```
[Teradata JDBC Driver] [Error 1536] [SQLState HY000] Invalid connection
parameter name path
```
`SparkFileOutputHandler` compounded it: it rejected the transform outright
when no file path was configured, so the user had to invent a path for a sink
that has none — and that invented value is precisely what got sent to the
driver.
### The fix
`isPathless(format)` names the formats whose sink is addressed entirely
through options rather than a path. Writing one of those now calls the
no-argument `save()`, so nothing enters the writer options:
```java
if (pathless) {
writer.save();
} else {
writer.save(path);
}
```
Only `jdbc` is in that set today. It is a named set rather than an `equals`
check so other option-addressed sinks can join it, but I deliberately did not
add `kafka`/`console`/`noop` speculatively — `jdbc` is the one with a reported
failure.
Also: the error message for a pathless format no longer quotes a path that
played no part in the write, and `SparkFileOutput` no longer demands a file
path when the format is pathless.
`SparkLakeTableSupport` shares `writeDataset` but only ever passes `delta`
or `iceberg`, both path-based, so it is unaffected.
### Tests
The obvious test — write to an embedded database and see whether it
complains — **does not work, and it is worth saying why.** I tried it with H2
first and it passed *with the fix reverted*: H2, like Postgres and MySQL,
silently ignores connection properties it does not recognise. Only a strict
driver such as Teradata's surfaces this, and depending on one in a unit test is
not an option.
So the tests assert the thing that is actually wrong — what Spark hands the
driver. `CapturingJdbcDriver` records the `Properties` of every connection and
delegates to H2, and the test asserts `path` is not among them. Reverting the
fix fails it with exactly the right message:
```
Spark sent 'path' as a connection property: [path] ==> expected: <false> but
was: <true>
```
`CapturingJdbcDriver` is top-level rather than nested on purpose: Spark's
`DriverRegistry` matches a registered driver by *canonical* name, which for a
nested class does not equal the binary name it is loaded by, and the lookup
fails with an internal error.
H2 is added in test scope only, with the version imported from
`hop-libs-jdbc` the same way `plugins/actions/waitforsql` does it.
Verified locally on JDK 21: `hop-engines-spark` **189 tests, 0 failures**.
`spotless:check` and `apache-rat:check` clean.
### Note on scope
I could not reproduce against Teradata itself — no access to that driver or
a Vantage instance. What is verified here is that Hop no longer sends `path`,
which is the cause identified in the report; @tmortada, confirmation against
your Teradata setup would be worth having before this is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
