gaogaotiantian commented on code in PR #57611:
URL: https://github.com/apache/spark/pull/57611#discussion_r3686283431
##########
python/pyspark/sql/connect/streaming/readwriter.py:
##########
@@ -510,13 +510,15 @@ def partitionBy(self, *cols: str) -> "DataStreamWriter":
...
@overload
def partitionBy(self, __cols: List[str]) -> "DataStreamWriter": ...
- def partitionBy(self, *cols: str) -> "DataStreamWriter": # type:
ignore[misc]
+ def partitionBy(self, *cols: Union[str, List[str]]) -> "DataStreamWriter":
Review Comment:
This is a very common issue in pyspark and we should deal with it at
repo-level in a proper way.
The current annotation is still incorrect because the function accepts
`tuple` in the logic below, but it is not listed in the annotation. Could you
refer to the APIs I changed in `dataframe.py`? And maybe make it consistent?
##########
python/pyspark/sql/connect/readwriter.py:
##########
@@ -130,15 +130,15 @@ def load(
self.schema(schema)
self.options(**options)
- paths = path
- if isinstance(path, str):
- paths = [path]
+ paths: Optional[List[str]] = None
+ if path is not None:
Review Comment:
This is a bit convoluted.
```python
paths: Optional[List[str]]
if isinstance(path, str):
paths = [path]
else:
paths = path
```
##########
python/pyspark/sql/connect/readwriter.py:
##########
@@ -57,7 +57,7 @@
class OptionUtils:
def _set_opts(
- self,
+ self: "SupportsOption",
Review Comment:
I think the better way here is to make `option` an abstract method of
`OptionUtils`. We don't really need a protocol here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
