[
https://issues.apache.org/jira/browse/SPARK-54951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tian Gao updated SPARK-54951:
-----------------------------
Summary: Make mypy pass with numpy stubs (was: Make mypy pass with scipy)
> Make mypy pass with numpy stubs
> -------------------------------
>
> Key: SPARK-54951
> URL: https://issues.apache.org/jira/browse/SPARK-54951
> Project: Spark
> Issue Type: Task
> Components: PySpark
> Affects Versions: 4.2.0
> Reporter: Tian Gao
> Priority: Major
>
> Currently even though mypy passes in CI, it fails locally. The reason is
> because CI does not install scipy which includes stub for numpy. scipy is
> part of requirements.txt so we installed it locally.
> The problem mypy shows is real - CI just failed to catch it.
> We can fix them all together but some of them are safer than the others. To
> avoid being reverted (and making it easier to review), we will fix them in
> different categorize.
> * Unused "type: ignore" comment - disable it in config because this line is
> sometimes used.
> * Obvious wrong code
> * Vague use of Iterable[float] - this is technically wrong. I think it was
> written like this because we were lazy. An Iterable[float] could be infinite,
> which can't be used as a vector. We should have a type alias for all types
> than can be converted to a vector.
> * Incompatible type due to scipy version - spmatrix is not the first citizen
> anymore I think. Many methods are not defined in this base class anymore. We
> can do some assertion, but this is not the safest change.
> * Incompatible type due to numpy version - this is a bit tricky. We had an
> effort to enforce numpy >= 2.0 but it got reverted. type hint is not super
> smart about this kind of situation. We should probably explicitly ignore
> these with comments that we should fix it after we drop numpy 1.x.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
