Yikun commented on code in PR #37117:
URL: https://github.com/apache/spark/pull/37117#discussion_r918485337
##########
python/pyspark/ml/util.py:
##########
@@ -536,10 +536,8 @@ def __get_class(clazz: str) -> Type[RL]:
"""
parts = clazz.split(".")
module = ".".join(parts[:-1])
- m = __import__(module)
- for comp in parts[1:]:
- m = getattr(m, comp)
- return m
+ m = __import__(module, fromlist=[parts[-1])
Review Comment:
I also take a deep look on
[this](https://docs.python.org/3/library/functions.html#import__):
> When the name variable is of the form package.module, normally, the
top-level package (the name up till the first dot) is returned, not the module
named by name. However, when a non-empty fromlist argument is given, the module
named by name is returned.
such for `pyspark.ml.classification.LinearSVC`
```python
>>> clazz = "pyspark.ml.classification.LinearSVC";parts =
clazz.split(".");module = ".".join(parts[:-1])
>>> __import__(module)
<module 'pyspark' from '/Users/yikun/spark/python/pyspark/__init__.py'>
>>> __import__(module, fromlist=[parts[-1]])
<module 'pyspark.ml.classification' from
'/Users/yikun/spark/python/pyspark/ml/classification.py'>
```
and also return same results of wrong name like:
`pyspark.ml.classification.unknow`, `pyspark.ml.unknow.LinearSVC`
After this, import with non-empty fromlist works well, can also simplified
implementations and cleanup the mypy error:
```
python/pyspark/ml/util.py:542: error: Incompatible return value type (got
Module, expected "Type[RL]") [return-value]
```
So, I'm fine with this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
