sarutak opened a new pull request, #57163:
URL: https://github.com/apache/spark/pull/57163
### What changes were proposed in this pull request?
This PR move the `import numpy` in `stateful_processor_api_client.py` from
the module top level to a lazy import inside `_normalize_state_value`,
triggered on first call and cached in a module variable.
### Why are the changes needed?
SPARK-57760 moved `import numpy` to the top level of
`stateful_processor_api_client.py` for micro-optimization. However, this module
is loaded transitively by `import pyspark` via:
import pyspark
-> pyspark.sql
-> pyspark.sql.streaming
-> pyspark.sql.streaming.stateful_processor
-> pyspark.sql.streaming.stateful_processor_api_client
-> import numpy
This causes `test_import_spark_libraries` to fail because the test asserts
that `import pyspark` does not pull in any third-party packages other than py4j.
https://github.com/apache/spark/actions/runs/29027743228/job/86156618984
```
pyspark.tests.test_import_spark with python3.12 failed:
Running tests...
----------------------------------------------------------------------
test_import_spark_libraries
(pyspark.tests.test_import_spark.ImportSparkTest.test_import_spark_libraries)
We want to ensure "import pyspark" is fast. It matters when we spawn ...
FAIL (0.252s)
======================================================================
FAIL [0.252s]: test_import_spark_libraries
(pyspark.tests.test_import_spark.ImportSparkTest.test_import_spark_libraries)
We want to ensure "import pyspark" is fast. It matters when we spawn
----------------------------------------------------------------------
Traceback (most recent call last):
File "/__w/spark/spark/python/pyspark/tests/test_import_spark.py", line
61, in test_import_spark_libraries
self.fail(
AssertionError: Unexpected 3rd party package 'numpy' imported during 'import
pyspark'
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
`pyspark.tests.test_import_spark.ImportSparkTest.test_import_spark_libraries`
now passes.
### Was this patch authored or co-authored using generative AI tooling?
Kiro CLI / Claude
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
