This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 4d90c597ec9 [SPARK-44923][PYTHON][BUILD] Some directories should be
cleared when regenerating files
4d90c597ec9 is described below
commit 4d90c597ec9089394af7e580cb3ad46cf1179098
Author: panbingkun <pbk1...@gmail.com>
AuthorDate: Wed Aug 23 17:12:14 2023 +0800
[SPARK-44923][PYTHON][BUILD] Some directories should be cleared when
regenerating files
### What changes were proposed in this pull request?
The pr aims to fix some bug in regenerating pyspark docs in certain
scenarios.
### Why are the changes needed?
- The following error occurred while I was regenerating the pyspark
document.
<img width="1001" alt="image"
src="https://github.com/apache/spark/assets/15246973/548abd63-4349-4267-b1fe-a293bd1e7f3e";>
- We can simply reproduce this problem as follows:
1.git reset --hard 3f380b9ecc8b27f6965b554061572e0990f0513
<img width="1416" alt="image"
src="https://github.com/apache/spark/assets/15246973/5ab9c8fc-5835-4ced-8d92-9d5e020b262a";>
2.make clean html, (at this point, it is successful.)
<img width="1000" alt="image"
src="https://github.com/apache/spark/assets/15246973/5c3ce07f-cbe8-4177-ae22-b16c3fc62e01";>
3.git pull, (at this point, the function `chr` has been deleted, but the
previously generated file(`pyspark.sql.functions.chr.rst`) will not be deleted.)
4.make clean html, (at this point, it is failed.)
<img width="1001" alt="image"
src="https://github.com/apache/spark/assets/15246973/548abd63-4349-4267-b1fe-a293bd1e7f3e";>
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
1.Pass GA.
2.Manually test.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #42622 from panbingkun/SPARK-44923.
Authored-by: panbingkun <pbk1...@gmail.com>
Signed-off-by: Ruifeng Zheng <ruife...@apache.org>
---
python/docs/source/conf.py | 26 ++++++++++----------------
1 file changed, 10 insertions(+), 16 deletions(-)
diff --git a/python/docs/source/conf.py b/python/docs/source/conf.py
index 0f57cb37cee..e4ca39d4f0c 100644
--- a/python/docs/source/conf.py
+++ b/python/docs/source/conf.py
@@ -33,22 +33,16 @@ generate_supported_api(output_rst_file_path)
# Remove previously generated rst files. Ignore errors just in case it stops
# generating whole docs.
-shutil.rmtree(
- "%s/reference/api" % os.path.dirname(os.path.abspath(__file__)),
ignore_errors=True)
-shutil.rmtree(
- "%s/reference/pyspark.pandas/api" %
os.path.dirname(os.path.abspath(__file__)),
- ignore_errors=True)
-try:
- os.mkdir("%s/reference/api" % os.path.dirname(os.path.abspath(__file__)))
-except OSError as e:
- if e.errno != errno.EEXIST:
- raise
-try:
- os.mkdir("%s/reference/pyspark.pandas/api" % os.path.dirname(
- os.path.abspath(__file__)))
-except OSError as e:
- if e.errno != errno.EEXIST:
- raise
+gen_rst_dirs = ["reference/api", "reference/pyspark.pandas/api",
+ "reference/pyspark.sql/api", "reference/pyspark.ss/api"]
+for gen_rst_dir in gen_rst_dirs:
+ absolute_gen_rst_dir = "%s/%s" %
(os.path.dirname(os.path.abspath(__file__)), gen_rst_dir)
+ shutil.rmtree(absolute_gen_rst_dir, ignore_errors=True)
+ try:
+ os.mkdir(absolute_gen_rst_dir)
+ except OSError as e:
+ if e.errno != errno.EEXIST:
+ raise
# -- General configuration ------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
