This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new e2e449e83cd [SPARK-40897][DOCS] Add some PySpark APIs to References e2e449e83cd is described below commit e2e449e83cdc743e7701ed93eae782ef9042f2d1 Author: Ruifeng Zheng <ruife...@apache.org> AuthorDate: Mon Oct 24 19:45:21 2022 +0900 [SPARK-40897][DOCS] Add some PySpark APIs to References ### What changes were proposed in this pull request? add following missing APIs to references: - StorageLevel.MEMORY_AND_DISK_DESER - TaskContext.cpus - BarrierTaskContext.cpus ### Why are the changes needed? they were missing in Reference ### Does this PR introduce _any_ user-facing change? Yes ### How was this patch tested? manually check, for `BarrierTaskContext.cpus` ``` In [10]: from pyspark import BarrierTaskContext In [11]: rdd = spark.sparkContext.parallelize([1]) In [12]: rdd.barrier().mapPartitions(lambda _: [BarrierTaskContext.get().cpus()]).collect() Out[12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1] ``` Closes #38373 from zhengruifeng/py_doc_missing. Authored-by: Ruifeng Zheng <ruife...@apache.org> Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> --- python/docs/source/reference/pyspark.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/python/docs/source/reference/pyspark.rst b/python/docs/source/reference/pyspark.rst index c3afae10ddb..ec3df071639 100644 --- a/python/docs/source/reference/pyspark.rst +++ b/python/docs/source/reference/pyspark.rst @@ -262,10 +262,12 @@ Management StorageLevel.DISK_ONLY_3 StorageLevel.MEMORY_AND_DISK StorageLevel.MEMORY_AND_DISK_2 + StorageLevel.MEMORY_AND_DISK_DESER StorageLevel.MEMORY_ONLY StorageLevel.MEMORY_ONLY_2 StorageLevel.OFF_HEAP TaskContext.attemptNumber + TaskContext.cpus TaskContext.get TaskContext.getLocalProperty TaskContext.partitionId @@ -277,6 +279,7 @@ Management BarrierTaskContext.allGather BarrierTaskContext.attemptNumber BarrierTaskContext.barrier + BarrierTaskContext.cpus BarrierTaskContext.get BarrierTaskContext.getLocalProperty BarrierTaskContext.getTaskInfos --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org