HyukjinKwon commented on a change in pull request #27466:
[SPARK-30722][PYTHON][DOCS] Update documentation for Pandas UDF with Python
type hints
URL: https://github.com/apache/spark/pull/27466#discussion_r375213650
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -364,7 +364,6 @@ def __hash__(self):
"pyspark.sql.avro.functions",
"pyspark.sql.pandas.conversion",
"pyspark.sql.pandas.map_ops",
- "pyspark.sql.pandas.functions",
Review comment:
All the tests in `pyspark.sql.pandas.functions` should be conditionally ran
- It should skip the tests if pandas or PyArrow are not available. However, we
have been skipping them always due to the lack of mechanism to conditionally
run the doctests.
Now, the doctests at `pyspark.sql.pandas.functions` have type hints that are
only for Python 3.5+. So, even if we skip all the tests like the previous way,
it shows compilation error due to illegal syntax in Python 2. This is why I had
to remove this from the module list to avoid compiling the doctests at all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
