hansva opened a new issue, #9557:
URL: https://github.com/apache/hop/issues/9557

   ### Apache Hop version?
   
   2.20.0
   
   ### Java version?
   
   21
   
   ### Operating system
   
   Other
   
   ### What happened?
   
   `IValueMeta.storeMetaInJson` / `loadMetaFromJson` (used by 
`JsonRowMeta.toJson` / `fromJson`) keep only a field's name, type, length, 
precision and conversion mask. Everything else a field carries is lost on the 
way through JSON and comes back as the default:
   
   | Lost | Effect after the round trip |
   |---|---|
   | origin | The transform a field comes from is empty, e.g. *Metadata 
structure* shows an empty Origin |
   | decimal, grouping and currency symbol | Numbers are formatted and parsed 
with the default symbols instead of the field's, e.g. `1.234,56` becomes 
`1,234.56` or fails to parse |
   | rounding type | Rounding falls back to the default |
   | trim type | Strings are no longer trimmed as configured |
   | date format locale, time zone and leniency | Dates are formatted and 
parsed in the JVM's locale and time zone, and strictness changes |
   | lenient string to number | Number parsing becomes strict or lenient 
depending on the default |
   | string encoding | Binary/string conversion uses the default encoding |
   | case insensitive, sorted descending | Comparisons and sorts ignore the 
field's case and order settings |
   | comments, collator settings, output padding | Lost as well |
   
   The binary format (`ValueMetaBase.writeMeta` / `readMeta`) keeps all of 
these, so the same row metadata behaves differently depending on which of the 
two formats it went through.
   
   ### Where this shows
   
   `JsonRowMeta` is how row metadata is handed over when it can't stay in 
memory:
   
   - **Beam engine:** every Hop transform that runs inside Beam gets its input 
and output row metadata as JSON (`TransformFn`, `TransformBatchTransform`, the 
Beam IO and group-by functions). A transform there formats and parses numbers 
and dates with the default symbols, locale and time zone instead of the 
field's, and compares without the field's case and sort settings. The same 
pipeline gives different results on the local engine and on Beam.
   - **Neo4j execution information location:** stores the row metadata of 
execution data samples as JSON. Samples read back in the execution viewer lose 
the field's formatting settings.
   - **Execution information transform:** writes row metadata as JSON into its 
output.
   
   ### Expected
   
   A field read back from JSON is the same as the field that was written, as 
with the binary format.
   
   ### Proposed fix
   
   In `ValueMetaBase.storeMetaInJson` / `loadMetaFromJson`, also write and read 
origin, comments, the decimal, grouping and currency symbols, rounding type, 
trim type, date format locale (language tag), time zone (ID) and leniency, 
lenient string to number, string encoding, case insensitive, the collator 
settings, sorted descending and output padding.
   
   Each key is only applied when present, so JSON written by older versions 
loads exactly as before, and older readers ignore the new keys.
   
   ### Tests
   
   `JsonRowMetaTest`: a row metadata with every one of these set to a 
non-default value survives `toJson` / `fromJson` unchanged. JSON without the 
new keys still loads, with the defaults.
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: API


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to