bamaer commented on code in PR #9558:
URL: https://github.com/apache/hop/pull/9558#discussion_r4625436761
##########
core/src/main/java/org/apache/hop/core/row/value/ValueMetaBase.java:
##########
@@ -6077,6 +6097,32 @@ public void storeMetaInJson(JSONObject jValue) throws
HopException {
jValue.put(CONST_LENGTH, getLength());
jValue.put(CONST_PRECISION, getPrecision());
jValue.put("conversionMask", getConversionMask());
+ // The rest of what the binary format (writeMeta) keeps: where the field
comes from and how
+ // its values are written, read and compared.
+ jValue.put(JSON_ORIGIN, getOrigin());
+ jValue.put(JSON_COMMENTS, getComments());
+ jValue.put(JSON_DECIMAL_SYMBOL, getDecimalSymbol());
+ jValue.put(JSON_GROUPING_SYMBOL, getGroupingSymbol());
+ jValue.put(JSON_CURRENCY_SYMBOL, getCurrencySymbol());
+ jValue.put(JSON_ROUNDING_TYPE, getRoundingType());
+ jValue.put(JSON_TRIM_TYPE, (long) getTrimType());
+ jValue.put(JSON_DATE_FORMAT_LENIENT, isDateFormatLenient());
+ if (getDateFormatLocale() != null) {
+ jValue.put(JSON_DATE_FORMAT_LOCALE,
getDateFormatLocale().toLanguageTag());
+ }
+ if (getDateFormatTimeZone() != null) {
+ jValue.put(JSON_DATE_FORMAT_TIME_ZONE, getDateFormatTimeZone().getID());
+ }
+ jValue.put(JSON_LENIENT_STRING_TO_NUMBER, isLenientStringToNumber());
+ jValue.put(JSON_STRING_ENCODING, getStringEncoding());
+ jValue.put(JSON_CASE_INSENSITIVE, isCaseInsensitive());
+ jValue.put(JSON_COLLATOR_DISABLED, isCollatorDisabled());
+ if (getCollatorLocale() != null) {
+ jValue.put(JSON_COLLATOR_LOCALE, getCollatorLocale().toLanguageTag());
Review Comment:
**Blocker:** the constructor takes the decimal, grouping and currency
symbols, the date locale, the time zone and the collator locale from the JVM,
so these are now written even when nobody set them. On Beam and Spark the JSON
is written on the machine that launches the pipeline and read on the workers.
The workers now use the launcher's locale and time zone instead of their own.
Written with `nl-BE` / `Europe/Brussels`, read with `en-US` / `UTC` (as on
Dataflow workers). Number with no format, Date with mask `yyyy-MM-dd HH:mm`:
| | main | this PR |
|---|---|---|
| `getString(1.5)` | `1.5` | `1,5` |
| `getString(2025-11-19 10:00 UTC)` | `2025-11-19 10:00` | `2025-11-19
11:00` |
Beam File Output writes values with `rowMeta.getString(...)`
(`HopToStringFn`). A pipeline launched from a Belgian or German machine writes
different files after the upgrade, and with a `,` separator and no enclosure,
`1,5` adds a column. The output also changes depending on which machine
launches the job.
Suggested fix: write the keys that come from the JVM only when they differ
from a fresh value meta's defaults:
```java
ValueMetaBase defaults = new ValueMetaBase(getName(), getType());
if (!Objects.equals(getDecimalSymbol(), defaults.getDecimalSymbol())) {
jValue.put(JSON_DECIMAL_SYMBOL, getDecimalSymbol());
}
// same for grouping and currency symbol, date locale, time zone, collator
locale
// and lenientStringToNumber (from a system property)
```
Settings a user sets explicitly (the `1.234,56` case in #9557) still
survive. Settings nobody set keep the reader's defaults, as on main. With this
change the cross-locale test gives main's output and the PR's tests still pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]