This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/spark-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 0cec0beaec Update PySpark Testing Webpage
0cec0beaec is described below
commit 0cec0beaece362985f3984a984ea0a131c81d3f9
Author: Amanda Liu <[email protected]>
AuthorDate: Thu Nov 2 14:03:17 2023 +0900
Update PySpark Testing Webpage
This PR is a followup to
[#43587](https://github.com/apache/spark/pull/43587) to update the PySpark
Testing Guidelines webpage links. This will remove external links in the guide.
Author: Amanda Liu <[email protected]>
Closes #486 from asl3/update-pyspark-testing-guide.
---
.../3.5.0/api/python/getting_started/testing_pyspark.html | 10 +++++-----
.../3.5.0/api/python/getting_started/testing_pyspark.ipynb | 14 +++++++-------
2 files changed, 12 insertions(+), 12 deletions(-)
diff --git a/site/docs/3.5.0/api/python/getting_started/testing_pyspark.html
b/site/docs/3.5.0/api/python/getting_started/testing_pyspark.html
index 0b46ae15bd..b58886cf79 100644
--- a/site/docs/3.5.0/api/python/getting_started/testing_pyspark.html
+++ b/site/docs/3.5.0/api/python/getting_started/testing_pyspark.html
@@ -495,7 +495,7 @@ div.rendered_html tbody tr:hover {
<div class="section" id="Testing-PySpark">
<h1>Testing PySpark<a class="headerlink" href="#Testing-PySpark"
title="Permalink to this headline">¶</a></h1>
<p>This guide is a reference for writing robust tests for PySpark code.</p>
-<p>To view the docs for PySpark test utils, see here. To see the code for
PySpark built-in test utils, check out the Spark repository here. To see the
JIRA board tickets for the PySpark test framework, see here.</p>
+<p>To view the docs for PySpark test utils, see <a class="reference external"
href="https://spark.apache.org/docs/latest/api/python/reference/pyspark.testing.html";>here</a>.</p>
<div class="section" id="Build-a-PySpark-Application">
<h2>Build a PySpark Application<a class="headerlink"
href="#Build-a-PySpark-Application" title="Permalink to this
headline">¶</a></h2>
<p>Here is an example for how to start a PySpark application. Feel free to
skip to the next section, “Testing your PySpark Application,” if you already
have an application you’re ready to test.</p>
@@ -620,9 +620,9 @@ div.rendered_html tbody tr:hover {
</div>
</div>
<div class="section" id="Option-2:-Using-Unit-Test">
-<h3>Option 2: Using <a class="reference external"
href="https://docs.python.org/3/library/unittest.html";>Unit Test</a><a
class="headerlink" href="#Option-2:-Using-Unit-Test" title="Permalink to this
headline">¶</a></h3>
+<h3>Option 2: Using Unit Test<a class="headerlink"
href="#Option-2:-Using-Unit-Test" title="Permalink to this headline">¶</a></h3>
<p>For more complex testing scenarios, you may want to use a testing
framework.</p>
-<p>One of the most popular testing framework options is unit tests. Let’s walk
through how you can use the built-in Python <code class="docutils literal
notranslate"><span class="pre">unittest</span></code> library to write PySpark
tests. For more information about the <code class="docutils literal
notranslate"><span class="pre">unittest</span></code> library, see here: <a
class="reference external"
href="https://docs.python.org/3/library/unittest.html";>https://docs.python.org/3/library/
[...]
+<p>One of the most popular testing framework options is unit tests. Let’s walk
through how you can use the built-in Python <code class="docutils literal
notranslate"><span class="pre">unittest</span></code> library to write PySpark
tests.</p>
<p>First, you will need a Spark session. You can use the <code class="docutils
literal notranslate"><span class="pre">@classmethod</span></code> decorator
from the <code class="docutils literal notranslate"><span
class="pre">unittest</span></code> package to take care of setting up and
tearing down a Spark session.</p>
<div class="nbinput nblast docutils container">
<div class="prompt highlight-none notranslate"><div
class="highlight"><pre><span></span>[15]:
@@ -679,8 +679,8 @@ div.rendered_html tbody tr:hover {
<p>When run, <code class="docutils literal notranslate"><span
class="pre">unittest</span></code> will pick up all functions with a name
beginning with “test.”</p>
</div>
<div class="section" id="Option-3:-Using-Pytest">
-<h3>Option 3: Using <a class="reference external"
href="https://docs.pytest.org/en/7.1.x/contents.html";>Pytest</a><a
class="headerlink" href="#Option-3:-Using-Pytest" title="Permalink to this
headline">¶</a></h3>
-<p>We can also write our tests with <code class="docutils literal
notranslate"><span class="pre">pytest</span></code>, which is one of the most
popular Python testing frameworks. For more information about <code
class="docutils literal notranslate"><span class="pre">pytest</span></code>,
see the docs here: <a class="reference external"
href="https://docs.pytest.org/en/7.1.x/contents.html";>https://docs.pytest.org/en/7.1.x/contents.html</a>.</p>
+<h3>Option 3: Using Pytest<a class="headerlink" href="#Option-3:-Using-Pytest"
title="Permalink to this headline">¶</a></h3>
+<p>We can also write our tests with <code class="docutils literal
notranslate"><span class="pre">pytest</span></code>, which is one of the most
popular Python testing frameworks.</p>
<p>Using a <code class="docutils literal notranslate"><span
class="pre">pytest</span></code> fixture allows us to share a spark session
across tests, tearing it down when the tests are complete.</p>
<div class="nbinput nblast docutils container">
<div class="prompt highlight-none notranslate"><div
class="highlight"><pre><span></span>[20]:
diff --git a/site/docs/3.5.0/api/python/getting_started/testing_pyspark.ipynb
b/site/docs/3.5.0/api/python/getting_started/testing_pyspark.ipynb
index 268ace0437..3934c11676 100644
--- a/site/docs/3.5.0/api/python/getting_started/testing_pyspark.ipynb
+++ b/site/docs/3.5.0/api/python/getting_started/testing_pyspark.ipynb
@@ -9,7 +9,7 @@
"\n",
"This guide is a reference for writing robust tests for PySpark code.\n",
"\n",
- "To view the docs for PySpark test utils, see here. To see the code for
PySpark built-in test utils, check out the Spark repository here. To see the
JIRA board tickets for the PySpark test framework, see here."
+ "To view the docs for PySpark test utils, see
[here](https://spark.apache.org/docs/latest/api/python/reference/pyspark.testing.html)."
]
},
{
@@ -190,10 +190,10 @@
"id": "c67be105-f6b1-4083-ad11-9e819331eae8",
"metadata": {},
"source": [
- "### Option 2: Using [Unit
Test](https://docs.python.org/3/library/unittest.html)\n",
+ "### Option 2: Using Unit Test\n",
"For more complex testing scenarios, you may want to use a testing
framework.\n",
"\n",
- "One of the most popular testing framework options is unit tests. Let’s
walk through how you can use the built-in Python `unittest` library to write
PySpark tests. For more information about the `unittest` library, see here:
https://docs.python.org/3/library/unittest.html. \n",
+ "One of the most popular testing framework options is unit tests. Let’s
walk through how you can use the built-in Python `unittest` library to write
PySpark tests. \n",
"\n",
"First, you will need a Spark session. You can use the `@classmethod`
decorator from the `unittest` package to take care of setting up and tearing
down a Spark session."
]
@@ -271,9 +271,9 @@
"id": "7d79e53d-cc1e-4fdf-a069-478337bed83d",
"metadata": {},
"source": [
- "### Option 3: Using
[Pytest](https://docs.pytest.org/en/7.1.x/contents.html)\n",
+ "### Option 3: Using Pytest\n",
"\n",
- "We can also write our tests with `pytest`, which is one of the most
popular Python testing frameworks. For more information about `pytest`, see the
docs here: https://docs.pytest.org/en/7.1.x/contents.html.\n";,
+ "We can also write our tests with `pytest`, which is one of the most
popular Python testing frameworks.\n",
"\n",
"Using a `pytest` fixture allows us to share a spark session across tests,
tearing it down when the tests are complete."
]
@@ -318,7 +318,7 @@
" {\"name\": \"Eve A.\", \"age\": 28}] \n",
" \n",
" # Create a Spark DataFrame\n",
- " original_df = spark.createDataFrame(sample_data)\n",
+ " original_df = spark_fixture.createDataFrame(sample_data)\n",
" \n",
" # Apply the transformation function from before\n",
" transformed_df = remove_extra_spaces(original_df, \"name\")\n",
@@ -328,7 +328,7 @@
" {\"name\": \"Bob T.\", \"age\": 35}, \n",
" {\"name\": \"Eve A.\", \"age\": 28}]\n",
" \n",
- " expected_df = spark.createDataFrame(expected_data)\n",
+ " expected_df = spark_fixture.createDataFrame(expected_data)\n",
"\n",
" assertDataFrameEqual(transformed_df, expected_df)"
]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
