HyukjinKwon opened a new pull request, #57158: URL: https://github.com/apache/spark/pull/57158
### What changes were proposed in this pull request? This PR makes `numpy` a lazy import again in `pyspark/sql/streaming/stateful_processor_api_client.py`. [SPARK-57760](https://issues.apache.org/jira/browse/SPARK-57760) hoisted `import numpy as np` to module level (to define `has_numpy` and `_normalize_state_value` once). However, that module sits on the eager `import pyspark` chain: ``` pyspark -> pyspark.sql -> pyspark.sql.context -> pyspark.sql.session -> pyspark.sql.dataframe -> pyspark.sql.streaming -> pyspark.sql.streaming.stateful_processor -> pyspark.sql.streaming.stateful_processor_api_client -> import numpy ``` so it caused `numpy` to be imported during `import pyspark`. The fix: - Resolve `has_numpy` via `importlib.util.find_spec("numpy")`, which detects numpy **without importing it**. - Import numpy lazily inside `_normalize_state_value` (cached in `sys.modules` after the first call, so the per-call cost is negligible). This restores the pre-SPARK-57760 behavior (numpy was previously imported lazily inside `_serialize_to_bytes`) while keeping SPARK-57760's benefit of defining `_normalize_state_value` only once. ### Why are the changes needed? `pyspark.tests.test_import_spark` (added in [SPARK-56242](https://issues.apache.org/jira/browse/SPARK-56242)) asserts that no 3rd party package is imported during `import pyspark`, to keep import fast. Since SPARK-57760 landed, it fails with: ``` AssertionError: Unexpected 3rd party package 'numpy' imported during 'import pyspark' ``` This breaks two scheduled CI jobs: - `Build / Python-only (Minimum dependencies of PySpark)` on **master** — https://github.com/apache/spark/actions/workflows/build_python_minimum.yml - `Build / Java21 (Scala 2.13, JDK 21)` on **branch-4.x** — https://github.com/apache/spark/actions/workflows/build_java21.yml?query=branch%3Abranch-4.x The offending commit is present on both master (`0af85eee8171`) and branch-4.x (`181acbd8db57`), so this needs to be backported to branch-4.x as well. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Verified `import pyspark` no longer imports numpy, `pyspark.tests.test_import_spark` passes, and `_normalize_state_value` still normalizes numpy scalars and nested containers (list/tuple/dict/namedtuple/Row) correctly. Also validated on GitHub Actions on a fork. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
