This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 07f70227f8b [SPARK-44859][SS] Fix incorrect property name in
structured streaming doc
07f70227f8b is described below
commit 07f70227f8bf81928d98101a88fd2885784451f5
Author: Liang-Chi Hsieh <vii...@gmail.com>
AuthorDate: Thu Aug 17 15:52:00 2023 -0700
[SPARK-44859][SS] Fix incorrect property name in structured streaming doc
### What changes were proposed in this pull request?
We found that one structured streaming property for asynchronous progress
tracking is not correct when comparing with codebase.
### Why are the changes needed?
Fix incorrect property name in structured streaming document.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
None, doc change only.
Closes #42544 from viirya/minor_doc_fix.
Authored-by: Liang-Chi Hsieh <vii...@gmail.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
docs/structured-streaming-programming-guide.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/structured-streaming-programming-guide.md
b/docs/structured-streaming-programming-guide.md
index 53d5919d4dc..dc25adbdfd3 100644
--- a/docs/structured-streaming-programming-guide.md
+++ b/docs/structured-streaming-programming-guide.md
@@ -4093,7 +4093,7 @@ The table below describes the configurations for this
feature and default values
| Option | Value | Default | Description |
|-------------|-----------------|------------|---------------------|
|asyncProgressTrackingEnabled|true/false|false|enable or disable asynchronous
progress tracking|
-|asyncProgressCheckpointingInterval|minutes|1|the interval in which we commit
offsets and completion commits|
+|asyncProgressTrackingCheckpointIntervalMs|millisecond|1000|the interval in
which we commit offsets and completion commits|
## Limitations
The initial version of the feature has the following limitations:
@@ -4114,7 +4114,7 @@ Also the following error message may be printed in the
driver logs:
The offset log for batch x doesn't exist, which is required to restart the
query from the latest batch x from the offset log. Please ensure there are two
subsequent offset logs available for the latest batch via manually deleting the
offset file(s). Please also ensure the latest batch for commit log is equal or
one batch earlier than the latest batch for offset log.
```
-This is caused by the fact that when async progress tracking is enabled, the
framework will not checkpoint progress for every batch as would be done if
async progress tracking is not used. To solve this problem simply re-enable
“asyncProgressTrackingEnabled” and set “asyncProgressCheckpointingInterval” to
0 and run the streaming query until at least two micro-batches have been
processed. Async progress tracking can be now safely disabled and restarting
query should proceed normally.
+This is caused by the fact that when async progress tracking is enabled, the
framework will not checkpoint progress for every batch as would be done if
async progress tracking is not used. To solve this problem simply re-enable
“asyncProgressTrackingEnabled” and set
“asyncProgressTrackingCheckpointIntervalMs” to 0 and run the streaming query
until at least two micro-batches have been processed. Async progress tracking
can be now safely disabled and restarting query should proceed normally.
# Continuous Processing
## [Experimental]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
