This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 80374dc2a548 [MINOR][DOC] Fix typos and add missing references to
standalone and YARN deployment docs
80374dc2a548 is described below
commit 80374dc2a548d92b123018ba6b3a506e434a59f2
Author: Nicholas Chammas <[email protected]>
AuthorDate: Thu Aug 6 21:30:01 2026 +0800
[MINOR][DOC] Fix typos and add missing references to standalone and YARN
deployment docs
### What changes were proposed in this pull request?
Miscellaneous typo fixes to the standalone and YARN deployment docs.
Add references to the standalone docs that are missing, like for Kubernetes
and various `sbin` scripts.
### Why are the changes needed?
Just to improve the flow of these particular docs and add handy references
a reader would expect to find.
### Does this PR introduce _any_ user-facing change?
Documentation only.
### How was this patch tested?
I built the docs using `SKIP_API=1 bundle exec jekyll serve` and reviewed
them in my browser.
### Was this patch authored or co-authored using generative AI tooling?
GitHub Copilot helped find some of the typos.
Closes #57536 from nchammas/deployment-docs.
Authored-by: Nicholas Chammas <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
docs/running-on-yarn.md | 2 +-
docs/spark-standalone.md | 15 ++++++++++-----
2 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index f7f356822f63..bd82a8445388 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -773,7 +773,7 @@ Stage level scheduling is supported on YARN:
- When dynamic allocation is disabled: It allows users to specify different
task resource requirements at the stage level and will use the same executors
requested at startup.
- When dynamic allocation is enabled: It allows users to specify task and
executor resource requirements at the stage level and will request the extra
executors.
-One thing to note that is YARN specific is that each ResourceProfile requires
a different container priority on YARN. The mapping is simply the
ResourceProfile id becomes the priority, on YARN lower numbers are higher
priority. This means that profiles created earlier will have a higher priority
in YARN. Normally this won't matter as Spark finishes one stage before starting
another one, the only case this might have an affect is in a job server type
scenario, so its something to keep in mind.
+One thing to note that is YARN specific is that each ResourceProfile requires
a different container priority on YARN. The mapping is simply the
ResourceProfile id becomes the priority, on YARN lower numbers are higher
priority. This means that profiles created earlier will have a higher priority
in YARN. Normally this won't matter as Spark finishes one stage before starting
another one, the only case this might have an effect is in a job server type
scenario, so it's something to keep in mind.
Note there is a difference in the way custom resources are handled between the
base default profile and custom ResourceProfiles. To allow for the user to
request YARN containers with extra resources without Spark scheduling on them,
the user can specify resources via the
<code>spark.yarn.executor.resource.</code> config. Those configs are only used
in the base default profile though and do not get propagated into any other
custom ResourceProfiles. This is because there would be no way to [...]
# Important notes
diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md
index ec1656b0348c..b35e74d246ad 100644
--- a/docs/spark-standalone.md
+++ b/docs/spark-standalone.md
@@ -21,7 +21,7 @@ license: |
* This will become a table of contents (this text will be scraped).
{:toc}
-In addition to running on the YARN cluster manager, Spark also provides a
simple standalone deploy mode. You can launch a standalone cluster either
manually, by starting a master and workers by hand, or use our provided [launch
scripts](#cluster-launch-scripts). It is also possible to run these daemons on
a single machine for testing.
+In addition to running on the [YARN cluster manager](running-on-yarn.html) and
[Kubernetes](running-on-kubernetes.html), Spark also provides a simple
standalone deploy mode. You can launch a standalone cluster either manually, by
starting a master and workers by hand, or use our provided [launch
scripts](#cluster-launch-scripts). It is also possible to run these daemons on
a single machine for testing.
# Security
@@ -100,13 +100,16 @@ Once you've set up this file, you can launch or stop your
cluster with the follo
- `sbin/start-master.sh` - Starts a master instance on the machine the script
is executed on.
- `sbin/start-workers.sh` - Starts a worker instance on each machine specified
in the `conf/workers` file.
- `sbin/start-worker.sh` - Starts a worker instance on the machine the script
is executed on.
-- `sbin/start-connect-server.sh` - Starts a Spark Connect server on the
machine the script is executed on.
+- `sbin/start-connect-server.sh` - Starts a [Spark
Connect](spark-connect-overview.html) server on the machine the script is
executed on.
+- `sbin/start-history-server.sh` - Starts the History Server, which lets you
view logs for completed applications. Requires event logging to be enabled (see
[Monitoring and Instrumentation](monitoring.html)).
- `sbin/start-all.sh` - Starts both a master and a number of workers as
described above.
- `sbin/stop-master.sh` - Stops the master that was started via the
`sbin/start-master.sh` script.
- `sbin/stop-worker.sh` - Stops all worker instances on the machine the script
is executed on.
- `sbin/stop-workers.sh` - Stops all worker instances on the machines
specified in the `conf/workers` file.
- `sbin/stop-connect-server.sh` - Stops all Spark Connect server instances on
the machine the script is executed on.
+- `sbin/stop-history-server.sh` - Stops the History Server.
- `sbin/stop-all.sh` - Stops both the master and the workers as described
above.
+- `sbin/decommission-worker.sh` - Gracefully decommissions a worker, allowing
in-progress tasks to finish and shuffles to be migrated before the worker exits.
Note that these scripts must be executed on the machine you want to run the
Spark master on, not your local machine.
@@ -687,7 +690,7 @@ configurations, <code>curl</code> CLI command can provide
the required header li
```bash
$ curl -XPOST http://IP:PORT/v1/submissions/create \
---header "Authorization: Bearer
USER-PROVIDED-WEB-TOEN-SIGNED-BY-THE-SAME-SHARED-KEY"
+--header "Authorization: Bearer
USER-PROVIDED-WEB-TOKEN-SIGNED-BY-THE-SAME-SHARED-KEY"
...
```
@@ -748,7 +751,7 @@ worker during one single schedule iteration.
Stage level scheduling is supported on Standalone:
- When dynamic allocation is disabled: It allows users to specify different
task resource requirements at the stage level and will use the same executors
requested at startup.
-- When dynamic allocation is enabled: Currently, when the Master allocates
executors for one application, it will schedule based on the order of the
ResourceProfile ids for multiple ResourceProfiles. The ResourceProfile with
smaller id will be scheduled firstly. Normally this won’t matter as Spark
finishes one stage before starting another one, the only case this might have
an affect is in a job server type scenario, so its something to keep in mind.
For scheduling, we will only take exe [...]
+- When dynamic allocation is enabled: Currently, when the Master allocates
executors for one application, it will schedule based on the order of the
ResourceProfile ids for multiple ResourceProfiles. The ResourceProfile with
smaller id will be scheduled first. Normally this won’t matter as Spark
finishes one stage before starting another one; the only case this might have
an effect is in a job server type scenario, so it’s something to keep in mind.
For scheduling, we will only take exec [...]
## Caveats
@@ -760,6 +763,8 @@ Spark's standalone mode offers a web-based user interface
to monitor the cluster
In addition, detailed log output for each job is also written to the work
directory of each worker node (`SPARK_HOME/work` by default). You will see two
files for each job, `stdout` and `stderr`, with all output it wrote to its
console.
+To track and review logs across completed applications, [enable event logging
and start the History Server](monitoring.html#viewing-after-the-fact).
+
# Running Alongside Hadoop
@@ -835,7 +840,7 @@ In order to enable this recovery mode, you can set
SPARK_DAEMON_JAVA_OPTS in spa
<td><code>spark.deploy.recoveryDirectory</code></td>
<td>""</td>
<td>The directory in which Spark will store recovery state, accessible
from the Master's perspective.
- Note that the directory should be clearly manually if
<code>spark.deploy.recoveryMode</code>
+ Note that the directory should be cleared manually if
<code>spark.deploy.recoveryMode</code>
or <code>spark.deploy.recoveryCompressionCodec</code> is changed.
</td>
<td>0.8.1</td>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
