Yicong-Huang opened a new pull request, #54010: URL: https://github.com/apache/spark/pull/54010
### What changes were proposed in this pull request? This PR extends the PyArrow array cast tests to cover both `safe=True` and `safe=False` behaviors in each test case. Changes: 1. **Modified `_run_cast_tests` method**: Now tests both `safe=True` and `safe=False` modes for each test case 2. **Extended test case format**: Support 3-tuple format `(src_arr, expected_safe_true, expected_safe_false)` where `expected_safe_false=None` means same behavior as `safe=True` 3. **Updated 200+ test cases**: All overflow/truncation cases now explicitly specify the `safe=False` behavior (wrapping, truncation, or saturation) Categories of `safe=False` behavior covered: - **Integer overflow**: Negative-to-unsigned wraps (e.g., `-1` → `255` for uint8) - **Integer narrowing**: Value wraps on overflow (e.g., `128` as int16 → `-128` as int8) - **Float-to-integer**: Truncation and saturation for out-of-range values - **Timestamp/Duration precision loss**: Truncation when casting to coarser units - **Decimal overflow**: Wrapping behavior for values exceeding integer limits ### Why are the changes needed? Part of [SPARK-54936](https://issues.apache.org/jira/browse/SPARK-54936). The existing tests only covered `safe=True` behavior. This PR adds comprehensive coverage for `safe=False` to ensure PySpark correctly handles PyArrow's overflow/truncation semantics, which is important for data processing pipelines that need to handle edge cases gracefully. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? All 39 tests in `PyArrowNumericalCastTests` pass with the updated test framework. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
