AvinashYerra opened a new pull request, #1431:
URL: https://github.com/apache/hamilton/pull/1431
This PR implements NDJSON (newline-delimited JSON) reader and writer support
for Polars, following the same pattern as the existing JSON reader/writer
implementation.
## Changes
- Added PolarsNDJSONReader class in
hamilton/plugins/polars_post_1_0_0_extensions.py
- Added PolarsNDJSONWriter class in
hamilton/plugins/polars_post_1_0_0_extensions.py
- Supports writing NDJSON files using data.write_ndjson()
- Registered both classes in `register_data_loaders()` function to make them
available through the materialization system
- Added the following tests:
- test_polars_ndjson() in tests/plugins/test_polars_extensions.py for
DataFrame support
- test_polars_ndjson() in
tests/plugins/test_polars_lazyframe_extensions.py for LazyFrame support
- Added example in examples/polars/materialization/my_script.py
## How I tested this
Ran the specific NDJSON tests:
- pytest tests/plugins/test_polars_extensions.py::test_polars_ndjson -v
- pytest
tests/plugins/test_polars_lazyframe_extensions.py::test_polars_ndjson -v
Both tests passed
## Notes
## Checklist
- [Y] PR has an informative and human-readable title (this will be pulled
into the release notes)
- [ ] Changes are limited to a single goal (no scope creep)
- [Y] Code passed the pre-commit check & code is left cleaner/nicer than
when first encountered.
- [ ] Any _change_ in functionality is tested
- [ ] New functions are documented (with a description, list of inputs, and
expected output)
- [ ] Placeholder code is flagged / future TODOs are captured in comments
- [ ] Project documentation has been updated if adding/changing
functionality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
