nicklamiller commented on code in PR #47493:
URL: https://github.com/apache/spark/pull/47493#discussion_r1736903772
##########
python/pyspark/sql/_typing.pyi:
##########
@@ -26,7 +26,7 @@ from typing import (
TypeVar,
Union,
)
-from typing_extensions import Literal, Protocol
+from typing_extensions import Concatenate, Literal, ParamSpec, Protocol
Review Comment:
You're right that `Concatenate` isn't available from `typing` in Python 3.9;
however, `typing_extensions` backports this functionality for Python 3.9:
https://github.com/python/typing_extensions/blob/d9509f902c20e3d51c62b8abe522809b4760c0ff/src/typing_extensions.py#L1815-L1833
```python
# 3.10+
if hasattr(typing, 'Concatenate'):
Concatenate = typing.Concatenate
_ConcatenateGenericAlias = typing._ConcatenateGenericAlias
# 3.9
elif sys.version_info[:2] >= (3, 9):
@_ExtensionsSpecialForm
def Concatenate(self, parameters):
"""Used in conjunction with ``ParamSpec`` and ``Callable`` to
represent a
higher order function which adds, removes or transforms parameters
of a
callable.
For example::
Callable[Concatenate[int, P], int]
See PEP 612 for detailed information.
"""
return _concatenate_getitem(self, parameters)
```
To confirm there's no issues with Python 3.9, I've built Spark with these
changes and `transform` still works with and without `typing_extensions`
installed in a fresh Python 3.9 virtual environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
