Abhi2939 opened a new pull request, #41154:
URL: https://github.com/apache/beam/pull/41154
Fixes #41102
`WriteToJson` writes through `DataFrame.to_json`, and pandas defaults to
`double_precision=10`, so every float was silently rounded to 10 decimal
places. Values smaller than about 1e-10 could be wiped out entirely.
This PR defaults `double_precision` to 15 (the pandas maximum) when the
caller doesn't pass it. A caller who passes `double_precision` explicitly gets
exactly what they asked for.
**Before** (same pipeline on master, Beam YAML and plain Python give
identical output):
```
{"id":1,"x":0.123456789}
{"id":2,"x":3.3333333333}
{"id":3,"x":0.0}
```
for inputs `0.1234567890123`, `3.3333333333333335` and `1e-11`.
**After:**
```
{"id":1,"x":0.1234567890123}
{"id":2,"x":3.333333333333334}
{"id":3,"x":0.00000000001}
```
**Known limitation.** `double_precision` is capped at 15 in pandas, so
floats that need 16-17 significant digits to round-trip (for example
`3.3333333333333335` or `0.1 + 0.2`) can still be rounded. I noted this in the
docstring. Exact round-tripping would need `WriteToJson` to stop using
`DataFrame.to_json` for the `records`/`lines=True` case (for example by writing
with `json.dumps`). That touches the shared `_WriteToPandasFileSink` and has
parity gaps with the pandas output (timestamps written as epoch ms, NaT/NaN
written as null), so I kept it out of this PR. Happy to follow up if
maintainers want it.
**Changes**
- `textio.py`: `kwargs.setdefault('double_precision', 15)` in `WriteToJson`,
plus a docstring note.
- `textio_test.py`: two tests in `JsonTest`:
- the default no longer truncates (`0.1234567890123` and `1e-11` are
preserved; this test fails without the fix)
- an explicit `double_precision=10` is still honored.
- `CHANGES.md`: bugfix entry under 2.79.0.
**Testing**
- `pytest apache_beam/io/textio_test.py -k json`
- `pytest apache_beam/yaml/yaml_transform_test.py
apache_beam/cli/main_test.py`
- `yapf --diff` is clean (yapf 0.43.0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
