hansva opened a new issue, #9119: URL: https://github.com/apache/hop/issues/9119
### What would you like to happen? ### Summary Since 2.18 a saved `<format/>` (or a blank JSON value) loads as `""` instead of `null`. Many transforms still guard overrides with `if (x != null) vm.setX(x)`. A blank Format, Decimal, Grouping or Currency setting therefore overwrites the incoming field's metadata with an empty string. #8956 fixed this in Text File Output's `getFields()`. An audit of all 277 transforms found the same pattern, and related ones, in many other places. Of 34 suspects, 26 are bugs, 3 are inconsistencies and 5 are by design. ### Root causes and proposed fixes Most of the bugs come from four root causes, so a few core helpers close most of them: | | Root cause | Fix | Closes | |---|---|---|---| | A | A blank override overwrites inherited metadata | One `applyOverrides(vm, …)` helper with `Utils.isEmpty` / `>= 0` guards | Text File Output, XML Output, Select values (rounding, length), Concat fields | | B | Text is parsed with a bare `ValueMetaString` or a hand-rolled parser, so it follows the JVM locale | Parse with `target.cloneToType(String)`. Without a format column, use a locale-neutral "machine" meta (`.` decimal, ISO dates) | Database lookup, Stream lookup, Value mapper, Null if, Add constants, OData, YAML, Google Sheets Input, Table Input | | C | `cloneValueMeta` on a type change copies storage and masks across type families, and drops currency and sort flags | On a type change: storage NORMAL, no mask between Number and Date, keep currency, collator locale and sort flags | Replace-mode ClassCastExceptions (Formula, Janino, Regex, JavaScript, …), Select values Number→Date, Merge join sort flags | | D | Machine formats (SQL literals, bulk-load files) are written with display formatting | A helper for neutral text (`toPlainString`, `.` decimal, ISO-8601 with offset) | SQL File Output, Execute SQL, Web services, Parquet, Snowflake, Redshift, CrateDB and PG bulk loaders, Iceberg, Salesforce Input | ### Most important cases - **Text File Output** clears decimal, grouping, currency, rounding, trim and length when they are left blank. - **Text File Input** date locale is broken for non-English locales (regression from 6bdc5fd095). - **Salesforce Input** turns `1234.56` into `123456` on `de_DE`. - **Select values**: the storage change does nothing (regression #1929), and a blank rounding setting is saved as `half_even`. - **Database lookup, Stream lookup, Value mapper, Null if**: default values are parsed in the JVM locale, so `1.5` becomes `15`. ### Plan 1. Fix root causes A and C first: they are small core changes and continue #8956. 2. Fix the Text File Input and Select values regressions. 3. Fix root cause D after a short design note on neutral formats. 4. Add an integration-test run in `de_DE` (or `nl_BE`). CI pins `en_US` and cannot catch locale regressions today. ### Compatibility Eight fixes are high risk because they change output that existing pipelines may rely on: Text File Output, XML Output, Add XML, Concat fields, Table Input, Execute SQL, Group by and Memory group by. For these, give new transforms the new behaviour. Transforms loaded from existing files keep the old behaviour behind a compatibility checkbox, or a `HOP_*` variable. Each change gets a before/after example in the release notes. ### Issue Priority Priority: 2 ### Issue Component Component: API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
