zhengruifeng opened a new pull request, #57781: URL: https://github.com/apache/spark/pull/57781
### What changes were proposed in this pull request? This PR adds a versioned internal serialization envelope for PySpark Classic broadcast variables. Native PyArrow Table, RecordBatch, Array, and ChunkedArray values use Arrow IPC streams. All other Python values continue to use pickle. The public Broadcast.dump, Broadcast.load, and Broadcast.load_from_path methods remain pickle-specific for compatibility. The Arrow import is also avoided for ordinary broadcasts unless PyArrow is already loaded. Tests cover Arrow type and value preservation, the pickle fallback, driver and worker reads, and encrypted broadcast transport. ### Why are the changes needed? PyArrow values already have a columnar representation. Serializing them through Arrow IPC avoids the Python object reconstruction cost of generic pickle and substantially reduces deserialization CPU for large Arrow-native broadcast values while keeping the existing fallback for general Python classes. ### Does this PR introduce _any_ user-facing change? Yes. In PySpark Classic, broadcasting a native PyArrow Table, RecordBatch, Array, or ChunkedArray now uses Arrow IPC internally. The value exposed on the driver and workers retains the same PyArrow type and data. Other broadcast values continue to use pickle. ### How was this patch tested? - Added codec round-trip and pickle-fallback unit tests. - Added driver and worker integration tests, including an encrypted transport variant. - build/sbt -Phive package passed. - The focused codec and unencrypted integration tests passed with Python 3.13 in the spark-dev-313 conda environment. - Ruff lint, Ruff format checking, git diff checking, line-length scanning, and non-ASCII scanning passed. The encrypted variant could not complete locally because the environment crashes in the OpenSSL FIPS self-test. The existing pickle-only BroadcastTest.test_broadcast_with_encryption reproduces the identical crash, so this is not caused by the Arrow serialization path. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
