[
https://issues.apache.org/jira/browse/SPARK-58073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-58073:
------------------------------------
Assignee: Hyukjin Kwon
> Avoid eagerly importing numpy in StatefulProcessorApiClient
> -----------------------------------------------------------
>
> Key: SPARK-58073
> URL: https://issues.apache.org/jira/browse/SPARK-58073
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Structured Streaming
> Affects Versions: 4.2.0, 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
>
> SPARK-57760 hoisted {{import numpy as np}} to module level in
> {{python/pyspark/sql/streaming/stateful_processor_api_client.py}} (to define
> {{has_numpy}} and {{_normalize_state_value}} once). That module is on the
> eager {{import pyspark}} chain (pyspark -> sql -> context -> session ->
> dataframe -> streaming -> stateful_processor ->
> stateful_processor_api_client), so it now imports numpy during {{import
> pyspark}}.
> This breaks {{pyspark.tests.test_import_spark}} (added in SPARK-56242), which
> asserts that no 3rd party package is imported during {{import pyspark}}:
> {code}
> AssertionError: Unexpected 3rd party package 'numpy' imported during 'import
> pyspark'
> {code}
> As a result, the following scheduled jobs fail:
> * {{Build / Python-only (Minimum dependencies of PySpark)}} on master
> * {{Build / Java21 (Scala 2.13, JDK 21)}} on branch-4.x
> The offending commit is present on both master (0af85eee8171) and branch-4.x
> (181acbd8db57), so the fix needs to be backported to branch-4.x.
> Fix: resolve {{has_numpy}} via {{importlib.util.find_spec("numpy")}} (which
> does not import numpy) and import numpy lazily inside
> {{_normalize_state_value}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
