HyukjinKwon opened a new pull request, #233: URL: https://github.com/apache/spark-connect-rust/pull/233
### What changes were proposed in this pull request? [SPARK-61039](https://issues.apache.org/jira/browse/SPARK-61039): `checkpoint` and `localCheckpoint` now send their `eager` and `storageLevel`, as PySpark Connect does. `df.checkpoint(eager)` and `df.localCheckpoint(eager, storageLevel)` send `eager`, and for a local checkpoint the storage level, in their `CheckpointCommand`. A checkpoint that is not eager is materialized when it is first used, and a local one is kept at the storage level given. The core takes them in the new `DataFrame::checkpoint_with` and `local_checkpoint_with`. `DataFrame.persist`, `localCheckpoint` and `Catalog.cacheTable` now convert a `StorageLevel` to its proto with one helper. ### Why are the changes needed? The client sent every checkpoint as eager, at the default storage level, whatever the arguments. ### Does this PR introduce _any_ user-facing change? Yes. These checkpoints follow their `eager` and `storageLevel`, as in PySpark 4.3.0. ### How was this patch tested? - A new offline test checks the `local`, `eager` and `storage_level` of the `CheckpointCommand`s that `checkpoint` and `localCheckpoint` send to a mock Connect server, as official PySpark 4.3.0 sends them. - An e2e test now checks a lazy local checkpoint on disk, and a lazy checkpoint, against the server. - The lib tests of both crates (568 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1800 passed, 8 skipped), and the live drop-in and Arrow suites (105 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
