lisancao commented on code in PR #708:
URL: https://github.com/apache/spark-website/pull/708#discussion_r4253428842
##########
streaming/index.md:
##########
@@ -7,101 +7,85 @@ subproject: Streaming
---
-<div class="jumbotron">
- <a
href="{{site.baseurl}}/docs/latest/structured-streaming-programming-guide.html">Spark
Structured Streaming</a> makes it easy to build streaming applications and
pipelines with the same and familiar Spark APIs.
-</div>
+[Spark Structured
Streaming]({{site.baseurl}}/docs/latest/structured-streaming-programming-guide.html)
makes it easy to build streaming applications and pipelines with the same and
familiar Spark APIs.
+Spark Structured Streaming has two different modes:
+- **Spark Real-time Mode (RTM):** Designed for sub-second, operational
workloads. RTM provides continuous, millisecond-level processing that is up to
100x faster than micro-batch mode, eliminating the need to maintain a separate
real-time engine.
+- **Spark Micro-batch Mode:** Designed for analytical pipelines and ETL.
Micro-batch mode processes data streams as a series of small batch jobs, a
highly cost-effective option for second- or minute-level latencies.
-<div class="row row-padded">
- <div class="col-md-7 col-sm-7">
- <h2>Easy to use</h2>
- <p>
- Spark Structured Streaming abstracts away complex streaming concepts
such as incremental processing, checkpointing, and watermarks
- so that you can build streaming applications and pipelines without
learning any new concepts or tools.
- </p>
- </div>
- <div class="col-md-5 col-sm-5 col-padded-top col-center">
-
- <div style="margin-top: 15px; text-align: left; display: inline-block;">
- <div class="code">
- spark<br/>
- .<span class="sparkop">readStream</span><br/>
- .<span class="sparkop">select</span>(<span
class="closure">$"value"</span>.cast(<span
class="closure">"string"</span>).alias(<span
class="closure">"jsonData"</span>))<br/>
- .<span class="sparkop">select</span>(from_json(<span
class="closure">$"jsonData"</span>,jsonSchema).alias(<span
class="closure">"payload"</span>))<br/>
- .<span class="sparkop">writeStream</span><br/>
- .<span class="sparkop">trigger</span>(<span
class="closure">"1 seconds"</span>)<br/>
- .<span class="sparkop">start</span>()
- </div>
- </div>
- </div>
-</div>
+Both modes share the exact same codebase, so you can shift between
cost-effective ETL and sub-second operational streaming with a single
configuration change.
-<div class="row row-padded">
- <div class="col-md-7 col-sm-7">
- <h2>Unified batch and streaming APIs</h2>
- <p>
- Spark Structured Streaming provides the same structured APIs (DataFrames
and Datasets) as Spark so that you don’t need to develop on or maintain two
different technology stacks for batch and streaming.
- In addition, unified APIs make it easy to migrate your existing batch
Spark jobs to streaming jobs.
- </p>
- </div>
- <div class="col-md-5 col-sm-5 col-padded-top col-center">
- <div style="width: 100%; max-width: 300px; display: inline-block;">
- <img src="{{site.baseurl}}/images/spark-unified-batch-and-streaming.png"
style="width: 100%; max-width: 300px;">
- </div>
+## Easy to use
+
+Spark Structured Streaming abstracts away complex streaming concepts such as
incremental processing, checkpointing, and watermarks so you can build
streaming applications without learning new tools. Shifting from minute-level
ETL to millisecond-scale operational streaming requires only a change to your
trigger.
Review Comment:
Fixed in 45a8ee8. Reworded the four spots (the intro, "Easy to use", the
Real-time Mode sample lead-in, and "Built for both analytical and operational
workloads") to state that Real-time Mode also requires the Update output mode
and supports a defined subset of sources, operators, and sinks, rather than
only a trigger change. Regenerated `site/streaming/index.html`.
##########
streaming/index.md:
##########
@@ -7,101 +7,85 @@ subproject: Streaming
---
-<div class="jumbotron">
- <a
href="{{site.baseurl}}/docs/latest/structured-streaming-programming-guide.html">Spark
Structured Streaming</a> makes it easy to build streaming applications and
pipelines with the same and familiar Spark APIs.
-</div>
+[Spark Structured
Streaming]({{site.baseurl}}/docs/latest/structured-streaming-programming-guide.html)
makes it easy to build streaming applications and pipelines with the same and
familiar Spark APIs.
+Spark Structured Streaming has two different modes:
+- **Spark Real-time Mode (RTM):** Designed for sub-second, operational
workloads. RTM provides continuous, millisecond-level processing that is up to
100x faster than micro-batch mode, eliminating the need to maintain a separate
real-time engine.
+- **Spark Micro-batch Mode:** Designed for analytical pipelines and ETL.
Micro-batch mode processes data streams as a series of small batch jobs, a
highly cost-effective option for second- or minute-level latencies.
-<div class="row row-padded">
- <div class="col-md-7 col-sm-7">
- <h2>Easy to use</h2>
- <p>
- Spark Structured Streaming abstracts away complex streaming concepts
such as incremental processing, checkpointing, and watermarks
- so that you can build streaming applications and pipelines without
learning any new concepts or tools.
- </p>
- </div>
- <div class="col-md-5 col-sm-5 col-padded-top col-center">
-
- <div style="margin-top: 15px; text-align: left; display: inline-block;">
- <div class="code">
- spark<br/>
- .<span class="sparkop">readStream</span><br/>
- .<span class="sparkop">select</span>(<span
class="closure">$"value"</span>.cast(<span
class="closure">"string"</span>).alias(<span
class="closure">"jsonData"</span>))<br/>
- .<span class="sparkop">select</span>(from_json(<span
class="closure">$"jsonData"</span>,jsonSchema).alias(<span
class="closure">"payload"</span>))<br/>
- .<span class="sparkop">writeStream</span><br/>
- .<span class="sparkop">trigger</span>(<span
class="closure">"1 seconds"</span>)<br/>
- .<span class="sparkop">start</span>()
- </div>
- </div>
- </div>
-</div>
+Both modes share the exact same codebase, so you can shift between
cost-effective ETL and sub-second operational streaming with a single
configuration change.
-<div class="row row-padded">
- <div class="col-md-7 col-sm-7">
- <h2>Unified batch and streaming APIs</h2>
- <p>
- Spark Structured Streaming provides the same structured APIs (DataFrames
and Datasets) as Spark so that you don’t need to develop on or maintain two
different technology stacks for batch and streaming.
- In addition, unified APIs make it easy to migrate your existing batch
Spark jobs to streaming jobs.
- </p>
- </div>
- <div class="col-md-5 col-sm-5 col-padded-top col-center">
- <div style="width: 100%; max-width: 300px; display: inline-block;">
- <img src="{{site.baseurl}}/images/spark-unified-batch-and-streaming.png"
style="width: 100%; max-width: 300px;">
- </div>
+## Easy to use
+
+Spark Structured Streaming abstracts away complex streaming concepts such as
incremental processing, checkpointing, and watermarks so you can build
streaming applications without learning new tools. Shifting from minute-level
ETL to millisecond-scale operational streaming requires only a change to your
trigger.
+
+**Micro-batch mode** — second- or minute-level latency for analytical
ETL:
+
+<div style="margin-top: 15px; text-align: left; display: inline-block;">
+ <div class="code">
+ spark<br/>
+ .<span class="sparkop">readStream</span><br/>
+ .<span class="sparkop">select</span>(<span
class="closure">$"value"</span>.cast(<span
class="closure">"string"</span>).alias(<span
class="closure">"jsonData"</span>))<br/>
+ .<span class="sparkop">select</span>(from_json(<span
class="closure">$"jsonData"</span>, jsonSchema).alias(<span
class="closure">"payload"</span>))<br/>
+ .<span class="sparkop">writeStream</span><br/>
+ .<span class="sparkop">trigger</span>(Trigger.<span
class="sparkop">ProcessingTime</span>(<span class="closure">"1
minute"</span>))<br/>
+ .<span class="sparkop">start</span>()
</div>
</div>
-<div class="row row-padded">
- <div class="col-md-7 col-sm-7">
- <h2>Low latency and cost effective</h2>
- <p>
- Spark Structured Streaming uses the same underlying architecture as
Spark so that you can take advantage of all the performance and cost
optimizations built into the Spark engine.
- With Spark Structured Streaming, you can build low latency streaming
applications and pipelines cost effectively.
- </p>
- </div>
- <div class="col-md-5 col-sm-5 col-padded-top col-center">
- <div style="width: 100%; max-width: 300px; display: inline-block;">
- <img
src="{{site.baseurl}}/images/spark-structured-streaming-incremental-execution.png"
style="width: 100%; max-width: 300px;">
- </div>
+**Real-time Mode** — millisecond-scale latency for operational
workloads. Switch the trigger (the output mode must be `update`); the rest of
the query is unchanged:
+
+<div style="margin-top: 15px; text-align: left; display: inline-block;">
+ <div class="code">
+ spark<br/>
+ .<span class="sparkop">readStream</span><br/>
+ .<span class="sparkop">select</span>(<span
class="closure">$"value"</span>.cast(<span
class="closure">"string"</span>).alias(<span
class="closure">"jsonData"</span>))<br/>
+ .<span class="sparkop">select</span>(from_json(<span
class="closure">$"jsonData"</span>, jsonSchema).alias(<span
class="closure">"payload"</span>))<br/>
+ .<span class="sparkop">writeStream</span><br/>
+ .<span class="sparkop">outputMode</span>(<span
class="closure">"update"</span>)<br/>
+ .<span class="sparkop">trigger</span>(Trigger.<span
class="sparkop">RealTime</span>(<span class="closure">"5 minutes"</span>))<br/>
+ .<span class="sparkop">start</span>()
</div>
</div>
-<div class="row">
-
- <div class="col-md-4 col-padded">
- <h3>Getting started</h3>
- <p>
- To get started with Spark Structured Streaming:
- </p>
- <ul class="list-narrow">
- <li><a href="{{site.baseurl}}/downloads.html">Download Spark</a>. It
includes Structured Streaming as a module.</li>
- <li>Read the <a
href="{{site.baseurl}}/docs/latest/structured-streaming-programming-guide.html">Spark
Structured Streaming programming guide</a>, which includes programming models,
tutorials, configurations, etc.</li>
- </ul>
- </div>
+## Unified batch and streaming APIs
- <div class="col-md-4 col-padded">
- <h3>Community</h3>
- <p>
- Spark Structured Streaming is developed as part of Apache Spark. It thus
gets tested and updated with each Spark release.
- </p>
- <p>
- If you have questions about the system, ask on the
- <a href="{{site.baseurl}}/community.html#mailing-lists">Spark mailing
lists</a>.
- </p>
- <p>
- The Spark Structured Streaming developers welcome contributions. If
you'd like to help out,
- read <a href="{{site.baseurl}}/contributing.html">how to
- contribute to Spark</a>, and send us a patch!
- </p>
- </div>
+Spark Structured Streaming provides the same structured APIs (DataFrames and
Datasets) as Spark, so you don't need to develop on or maintain two different
technology stacks for batch and streaming. The same codebase scales from
historical batch processing to real-time inference, and unified APIs make it
easy to migrate existing batch Spark jobs to streaming.
-</div>
+
-<div class="row">
- <div class="col-sm-12 col-center">
- <a href="{{site.baseurl}}/downloads.html" class="btn btn-cta btn-lg
btn-multiline">
- Download Apache Spark<br/><span class="small">Includes Spark Structured
Streaming</span>
- </a>
- </div>
-</div>
+## Built for both analytical and operational workloads
+
+Run cost-effective ETL and sub-second operational applications on the same
engine. Spark Structured Streaming supports both micro-batch and Real-time
execution, so you can change latency targets with a single-line trigger
configuration.
+
+- **Operational streaming (Spark Real-time Mode):** A continuous processing
architecture that executes events as they arrive. Real-time Mode is up to 100x
faster than micro-batch for operational streaming workloads, dropping
end-to-end latency to as low as a few milliseconds, enabling inline fraud
detection, live personalization, and real-time alerting without a separate
streaming engine.
+- **Analytical streaming (Spark Micro-batch Mode):** This model balances high
throughput with cost-efficiency. Use it for traditional ETL, Medallion
architectures, and ingestion pipelines where latency is measured in seconds or
minutes.
+
+## Low latency and cost-effective
+
+Spark Structured Streaming uses the same underlying architecture as Spark, so
you can take advantage of all the performance and cost optimizations built into
the Spark engine. While micro-batch execution provides highly cost-effective
processing for analytical ETL, Real-time Mode uses continuous processing with
pipelined and streaming shuffles to reach P99 latencies ranging from a few
milliseconds to around 300 ms depending on the workload. Run mission-critical
operational workloads with ultra-low latency without sacrificing exactly-once
guarantees.
+
+
Review Comment:
Fixed in 45a8ee8. Both diagrams now use an inline `<img ...
style="max-width: 100%; height: auto;">` with alt text, so they stay inside the
content column on narrow viewports. Regenerated the HTML.
##########
streaming/index.md:
##########
@@ -7,101 +7,85 @@ subproject: Streaming
---
-<div class="jumbotron">
- <a
href="{{site.baseurl}}/docs/latest/structured-streaming-programming-guide.html">Spark
Structured Streaming</a> makes it easy to build streaming applications and
pipelines with the same and familiar Spark APIs.
-</div>
+[Spark Structured
Streaming]({{site.baseurl}}/docs/latest/structured-streaming-programming-guide.html)
makes it easy to build streaming applications and pipelines with the same and
familiar Spark APIs.
+Spark Structured Streaming has two different modes:
+- **Spark Real-time Mode (RTM):** Designed for sub-second, operational
workloads. RTM provides continuous, millisecond-level processing that is up to
100x faster than micro-batch mode, eliminating the need to maintain a separate
real-time engine.
+- **Spark Micro-batch Mode:** Designed for analytical pipelines and ETL.
Micro-batch mode processes data streams as a series of small batch jobs, a
highly cost-effective option for second- or minute-level latencies.
-<div class="row row-padded">
- <div class="col-md-7 col-sm-7">
- <h2>Easy to use</h2>
- <p>
- Spark Structured Streaming abstracts away complex streaming concepts
such as incremental processing, checkpointing, and watermarks
- so that you can build streaming applications and pipelines without
learning any new concepts or tools.
- </p>
- </div>
- <div class="col-md-5 col-sm-5 col-padded-top col-center">
-
- <div style="margin-top: 15px; text-align: left; display: inline-block;">
- <div class="code">
- spark<br/>
- .<span class="sparkop">readStream</span><br/>
- .<span class="sparkop">select</span>(<span
class="closure">$"value"</span>.cast(<span
class="closure">"string"</span>).alias(<span
class="closure">"jsonData"</span>))<br/>
- .<span class="sparkop">select</span>(from_json(<span
class="closure">$"jsonData"</span>,jsonSchema).alias(<span
class="closure">"payload"</span>))<br/>
- .<span class="sparkop">writeStream</span><br/>
- .<span class="sparkop">trigger</span>(<span
class="closure">"1 seconds"</span>)<br/>
- .<span class="sparkop">start</span>()
- </div>
- </div>
- </div>
-</div>
+Both modes share the exact same codebase, so you can shift between
cost-effective ETL and sub-second operational streaming with a single
configuration change.
-<div class="row row-padded">
- <div class="col-md-7 col-sm-7">
- <h2>Unified batch and streaming APIs</h2>
- <p>
- Spark Structured Streaming provides the same structured APIs (DataFrames
and Datasets) as Spark so that you don’t need to develop on or maintain two
different technology stacks for batch and streaming.
- In addition, unified APIs make it easy to migrate your existing batch
Spark jobs to streaming jobs.
- </p>
- </div>
- <div class="col-md-5 col-sm-5 col-padded-top col-center">
- <div style="width: 100%; max-width: 300px; display: inline-block;">
- <img src="{{site.baseurl}}/images/spark-unified-batch-and-streaming.png"
style="width: 100%; max-width: 300px;">
- </div>
+## Easy to use
+
+Spark Structured Streaming abstracts away complex streaming concepts such as
incremental processing, checkpointing, and watermarks so you can build
streaming applications without learning new tools. Shifting from minute-level
ETL to millisecond-scale operational streaming requires only a change to your
trigger.
+
+**Micro-batch mode** — second- or minute-level latency for analytical
ETL:
+
+<div style="margin-top: 15px; text-align: left; display: inline-block;">
+ <div class="code">
+ spark<br/>
+ .<span class="sparkop">readStream</span><br/>
+ .<span class="sparkop">select</span>(<span
class="closure">$"value"</span>.cast(<span
class="closure">"string"</span>).alias(<span
class="closure">"jsonData"</span>))<br/>
+ .<span class="sparkop">select</span>(from_json(<span
class="closure">$"jsonData"</span>, jsonSchema).alias(<span
class="closure">"payload"</span>))<br/>
+ .<span class="sparkop">writeStream</span><br/>
+ .<span class="sparkop">trigger</span>(Trigger.<span
class="sparkop">ProcessingTime</span>(<span class="closure">"1
minute"</span>))<br/>
+ .<span class="sparkop">start</span>()
</div>
</div>
-<div class="row row-padded">
- <div class="col-md-7 col-sm-7">
- <h2>Low latency and cost effective</h2>
- <p>
- Spark Structured Streaming uses the same underlying architecture as
Spark so that you can take advantage of all the performance and cost
optimizations built into the Spark engine.
- With Spark Structured Streaming, you can build low latency streaming
applications and pipelines cost effectively.
- </p>
- </div>
- <div class="col-md-5 col-sm-5 col-padded-top col-center">
- <div style="width: 100%; max-width: 300px; display: inline-block;">
- <img
src="{{site.baseurl}}/images/spark-structured-streaming-incremental-execution.png"
style="width: 100%; max-width: 300px;">
- </div>
+**Real-time Mode** — millisecond-scale latency for operational
workloads. Switch the trigger (the output mode must be `update`); the rest of
the query is unchanged:
+
+<div style="margin-top: 15px; text-align: left; display: inline-block;">
+ <div class="code">
+ spark<br/>
+ .<span class="sparkop">readStream</span><br/>
+ .<span class="sparkop">select</span>(<span
class="closure">$"value"</span>.cast(<span
class="closure">"string"</span>).alias(<span
class="closure">"jsonData"</span>))<br/>
+ .<span class="sparkop">select</span>(from_json(<span
class="closure">$"jsonData"</span>, jsonSchema).alias(<span
class="closure">"payload"</span>))<br/>
+ .<span class="sparkop">writeStream</span><br/>
+ .<span class="sparkop">outputMode</span>(<span
class="closure">"update"</span>)<br/>
+ .<span class="sparkop">trigger</span>(Trigger.<span
class="sparkop">RealTime</span>(<span class="closure">"5 minutes"</span>))<br/>
+ .<span class="sparkop">start</span>()
</div>
</div>
-<div class="row">
-
- <div class="col-md-4 col-padded">
- <h3>Getting started</h3>
- <p>
- To get started with Spark Structured Streaming:
- </p>
- <ul class="list-narrow">
- <li><a href="{{site.baseurl}}/downloads.html">Download Spark</a>. It
includes Structured Streaming as a module.</li>
- <li>Read the <a
href="{{site.baseurl}}/docs/latest/structured-streaming-programming-guide.html">Spark
Structured Streaming programming guide</a>, which includes programming models,
tutorials, configurations, etc.</li>
- </ul>
- </div>
+## Unified batch and streaming APIs
- <div class="col-md-4 col-padded">
- <h3>Community</h3>
- <p>
- Spark Structured Streaming is developed as part of Apache Spark. It thus
gets tested and updated with each Spark release.
- </p>
- <p>
- If you have questions about the system, ask on the
- <a href="{{site.baseurl}}/community.html#mailing-lists">Spark mailing
lists</a>.
- </p>
- <p>
- The Spark Structured Streaming developers welcome contributions. If
you'd like to help out,
- read <a href="{{site.baseurl}}/contributing.html">how to
- contribute to Spark</a>, and send us a patch!
- </p>
- </div>
+Spark Structured Streaming provides the same structured APIs (DataFrames and
Datasets) as Spark, so you don't need to develop on or maintain two different
technology stacks for batch and streaming. The same codebase scales from
historical batch processing to real-time inference, and unified APIs make it
easy to migrate existing batch Spark jobs to streaming.
-</div>
+
-<div class="row">
- <div class="col-sm-12 col-center">
- <a href="{{site.baseurl}}/downloads.html" class="btn btn-cta btn-lg
btn-multiline">
- Download Apache Spark<br/><span class="small">Includes Spark Structured
Streaming</span>
- </a>
- </div>
-</div>
+## Built for both analytical and operational workloads
+
+Run cost-effective ETL and sub-second operational applications on the same
engine. Spark Structured Streaming supports both micro-batch and Real-time
execution, so you can change latency targets with a single-line trigger
configuration.
+
+- **Operational streaming (Spark Real-time Mode):** A continuous processing
architecture that executes events as they arrive. Real-time Mode is up to 100x
faster than micro-batch for operational streaming workloads, dropping
end-to-end latency to as low as a few milliseconds, enabling inline fraud
detection, live personalization, and real-time alerting without a separate
streaming engine.
+- **Analytical streaming (Spark Micro-batch Mode):** This model balances high
throughput with cost-efficiency. Use it for traditional ETL, Medallion
architectures, and ingestion pipelines where latency is measured in seconds or
minutes.
+
+## Low latency and cost-effective
+
+Spark Structured Streaming uses the same underlying architecture as Spark, so
you can take advantage of all the performance and cost optimizations built into
the Spark engine. While micro-batch execution provides highly cost-effective
processing for analytical ETL, Real-time Mode uses continuous processing with
pipelined and streaming shuffles to reach P99 latencies ranging from a few
milliseconds to around 300 ms depending on the workload. Run mission-critical
operational workloads with ultra-low latency without sacrificing exactly-once
guarantees.
+
+
+
+
+
+### Getting started
+
+To get started with Spark Structured Streaming:
+
+- [Download Spark]({{site.baseurl}}/downloads.html). It includes Structured
Streaming as a module.
+- Read the [Spark Structured Streaming programming
guide]({{site.baseurl}}/docs/latest/structured-streaming-programming-guide.html),
which includes programming models, tutorials, configurations, etc.
+
+### Community
+
+Spark Structured Streaming is developed as part of Apache Spark. It thus gets
tested and updated with each Spark release.
+
+If you have questions about the system, ask on the [Spark mailing
lists]({{site.baseurl}}/community.html#mailing-lists).
+
+The Spark Structured Streaming developers welcome contributions. If you'd like
to help out, read [how to contribute to
Spark]({{site.baseurl}}/contributing.html), and send us a patch!
+
+
+
+[Download Apache Spark]({{site.baseurl}}/downloads.html)
+[Includes Spark Structured Streaming]({{site.baseurl}}/downloads.html)
Review Comment:
Fixed in 45a8ee8. Restored the single captioned `btn btn-cta btn-lg
btn-multiline` button with the "Includes Spark Structured Streaming" caption
nested inside the anchor, in the centered row, matching the sql/mllib/graphx
pages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
