Hyukjin Kwon created SPARK-58072:
------------------------------------
Summary: Avoid eagerly importing numpy in
StatefulProcessorApiClient
Key: SPARK-58072
URL: https://issues.apache.org/jira/browse/SPARK-58072
Project: Spark
Issue Type: Bug
Components: Structured Streaming, PySpark, Tests
Affects Versions: 4.1.0
Reporter: Hyukjin Kwon
SPARK-57760 hoisted `import numpy as np` to module top-level in
`python/pyspark/sql/streaming/stateful_processor_api_client.py` (to define
`has_numpy` and `_normalize_state_value` once). However, that module is on the
eager `import pyspark` import chain:
{code}
pyspark -> pyspark.sql -> pyspark.sql.context -> pyspark.sql.session
-> pyspark.sql.dataframe -> pyspark.sql.streaming
-> pyspark.sql.streaming.stateful_processor
-> pyspark.sql.streaming.stateful_processor_api_client -> import numpy
{code}
As a result, `import pyspark` now eagerly imports numpy, which breaks
`pyspark.tests.test_import_spark.ImportSparkTest.test_import_spark_libraries`
(added in SPARK-56242). That test asserts no 3rd party package is imported
during `import pyspark` to keep import fast:
{code}
AssertionError: Unexpected 3rd party package 'numpy' imported during 'import
pyspark'
{code}
This fails several scheduled CI jobs that run the test with numpy installed,
e.g.:
- Build / Java25 (Scala 2.13, JDK 25) on master
- Build / RocksDB as UI Backend (JDK 17) on master
- Build / Python-only (Minimum dependencies of PySpark) on master
- Build / Java21 (Scala 2.13, JDK 21) on branch-4.x
The offending commit is present on both master and branch-4.x, so the fix
should be backported to branch-4.x as well.
Fix: resolve `has_numpy` via `importlib.util.find_spec("numpy")` (detects numpy
without importing it) and import numpy lazily inside `_normalize_state_value`.
This restores the pre-SPARK-57760 lazy-import behavior while keeping the
benefit of defining `_normalize_state_value` once.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
