HyukjinKwon commented on code in PR #42284:
URL: https://github.com/apache/spark/pull/42284#discussion_r1283880241
##########
python/docs/source/getting_started/testing_pyspark.ipynb:
##########
@@ -0,0 +1,525 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4ee2125b-f889-47e6-9c3d-8bd63a253683",
+ "metadata": {},
+ "source": [
+ "# Testing PySpark\n",
+ "\n",
+ "This guide is a reference for writing robust tests for PySpark code.\n",
+ "\n",
+ "To view the docs for PySpark test utils, see here. To see the code for
PySpark built-in test utils, check out the Spark repository here. To see the
JIRA board tickets for the PySpark test framework, see here."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e8ee4b6-9544-45e1-8a91-e71ed8ef8b9d",
+ "metadata": {},
+ "source": [
+ "## Build a PySpark Application\n",
+ "Here is an example for how to start a PySpark application. Feel free to
skip to the next section, “Testing your PySpark Application,” if you already
have an application you’re ready to test.\n",
+ "\n",
+ "First, start your Spark Session."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "64e82e7c",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'4.0.0.dev0'"
+ ]
+ },
+ "execution_count": 1,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import pyspark\n",
+ "pyspark.__version__"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "9af4a35b-17e8-4e45-816b-34c14c5902f7",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Setting default log level to \"WARN\".\n",
Review Comment:
I think you should remove this warning by setting
`spark.sparkContext.setLogLevel("FATAL")` first, and remove that cell
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
