mattcasters opened a new pull request, #8148: URL: https://github.com/apache/hop/pull/8148
## What Add a **Database value validation** transform that checks incoming rows against a target table *before* insert, so a PostgreSQL (or other RDBMS) failure is no longer "one column of one row among millions" with no useful field/value in the message. Fixes #8143 ## Why Table Output, Insert/Update and bulk COPY typically stop at the first database error. The message rarely names the stream field and the offending value. Data Validator can report all errors, but every rule is hand-authored and drifts from the live table. ## How - At start, load table column types, lengths, nullability, defaults and encoding, then disconnect. Row processing is CPU-only. - Every mapped field on every row is checked. All failures on a row are concatenated onto one error-handling hop (`DBV001`–`DBV008`). - Valid rows pass through unchanged. - Empty mapping matches stream fields to table columns by name. Checks (generic JDBC + PostgreSQL enricher): | Code | Meaning | |---|---| | DBV001 | Null in a NOT NULL column | | DBV002 | String too long (Unicode code points for `varchar`) | | DBV003 | Encoding / `NUL` (`U+0000`) | | DBV004 | `numeric(p,s)` overflow after rounding to scale | | DBV005 | Type conversion | | DBV006 | Integer out of range (`int2`/`int4`/`int8`) | | DBV007 | Invalid UUID | | DBV008 | Invalid JSON/JSONB | Unique keys, foreign keys and CHECK constraints are not pre-flighted. ## Tests - Unit: `ColumnValueValidatorTest` (11), `PostgreSqlColumnValueConstraintsTest` (6), `DatabaseValueValidationMetaTest` (load/save) - Integration: `main-0043-database-value-validation` (PostgreSQL `unit-test-db`) — mixed good/bad rows, golden error hop, only the valid row is inserted ## Docs User-manual page plus links from Table Output, Insert/Update, PostgreSQL bulk loader, Data Validator and error handling. ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Run `mvn clean install apache-rat:check` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [x] If you have a group of commits related to the same change, please squash your commits into one and force push your branch using `git rebase -i`. - [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. To make clear that you license your contribution under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0) you have to acknowledge this by using the following check-box. - [x] I hereby declare this contribution to be licensed under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0) - [ ] In any other case, please file an [Apache Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
