dbtsai opened a new pull request, #115: URL: https://github.com/apache/spark-connect-rust/pull/115
### What changes were proposed in this pull request? Accept `Value::Byte` and `Value::Short` in the shared floating-point Arrow encoder, using the same numeric cast policy as the existing Integer and Long inputs. The two new match arms also apply to nested array, map-value, and struct leaves. ### Why are the changes needed? [SPARK-60641](https://issues.apache.org/jira/browse/SPARK-60641): native Rust Rows collected from tinyint/smallint columns currently fail with `Type mismatch in row data` when reused with a Float or Double schema, although wider integer inputs already work. ### Does this PR introduce _any_ user-facing change? Yes. Rust `create_dataframe` accepts Byte/Short values for Float/Double fields, including nested fields. For example, a Row containing `Value::Short(7)` can be encoded with a Float field. This concerns the Rust Row API; Python explicit-schema verification rules are unchanged. ### How was this patch tested? - Added an input-to-IPC regression covering both targets, Byte/Short minima and maxima, negative values, zero, nulls, scalar/array/map-value/struct leaves, output Arrow types and values, and rejection of incompatible Boolean/String/Binary inputs. - Added and ran a live Spark Connect regression collecting tinyint/smallint Rows and recreating Float/Double DataFrames, checking result types and values. - All 387 Rust client library tests passed. All integration tests compiled in release mode, both with and without `wasm-udf`. Formatting and diff checks passed; the Python extension built successfully. - Official `test_parity_types.py` through the rebuilt Rust transport against a local Spark 4.2.0-SNAPSHOT server: **69 passed, 21 skipped**. Initial server configuration failures also reproduced with the reference client; the final run used Python 3.12 workers and enabled spatial support. - Python extension introspection retained the baseline's 108 classes and signatures. - Clippy passed for the library and changed integration target with the existing approximate-constant lint allowed. Unmodified all-target checks encounter existing approximate-constant errors elsewhere; workspace Clippy additionally requires the unavailable `wasm32-unknown-unknown` target. The complete official Connect suite was not run. This PR is a draft pending the remaining repository-wide checks. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
