HyukjinKwon opened a new pull request, #57169: URL: https://github.com/apache/spark/pull/57169
JIRA: https://issues.apache.org/jira/browse/SPARK-58071 ### What changes were proposed in this pull request? This PR makes `pyspark/sql/streaming/stateful_processor_api_client.py` resolve NumPy lazily instead of importing it at module load time. SPARK-57760 moved `import numpy as np` to the module top level (inside a `try/except ImportError` block). This module is transitively imported by `import pyspark` (`pyspark` -> `pyspark.sql` -> `pyspark.sql.streaming` -> `stateful_processor_api_client`), so NumPy is now imported during `import pyspark`. Instead of the top-level import, this PR: - adds a cached `has_numpy()` helper that performs the availability check on first use, and - imports `numpy` locally inside `_normalize_state_value` (only invoked when normalizing state values). The observable behavior is unchanged. ### Why are the changes needed? `import pyspark` is expected to stay fast and free of 3rd party libraries. `python/pyspark/tests/test_import_spark.py::ImportSparkTest::test_import_spark_libraries` enforces this and is currently failing on `master` / `branch-4.x`: ``` AssertionError: Unexpected 3rd party package 'numpy' imported during 'import pyspark' ``` This has been failing the scheduled Non-ANSI builds (e.g. `build_non_ansi.yml` on `branch-4.x`). ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - `python -X importtime -c "import pyspark"` no longer shows `numpy` in the import trace. - `python/pyspark/tests/test_import_spark.py` passes. - Verified `_normalize_state_value` still converts NumPy scalars and nested containers correctly, and the scalar fast-path is unchanged. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
