Github user zsxwing commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22627#discussion_r222818615
  
    --- Diff: docs/structured-streaming-programming-guide.md ---
    @@ -2709,6 +2935,78 @@ write.stream(aggDF, "memory", outputMode = 
"complete", checkpointLocation = "pat
     </div>
     </div>
     
    +
    +## Recovery Semantics after Changes in a Streaming Query
    +There are limitations on what changes in a streaming query are allowed 
between restarts from the 
    +same checkpoint location. Here are a few kinds of changes that are either 
not allowed, or 
    +the effect of the change is not well-defined. For all of them:
    +
    +- The term *allowed* means you can do the specified change but whether the 
semantics of its effect 
    +  is well-defined depends on the query and the change.
    +
    +- The term *not allowed* means you should not do the specified change as 
the restarted query is likely 
    +  to fail with unpredictable errors. `sdf` represents a streaming 
DataFrame/Dataset 
    +  generated with sparkSession.readStream.
    +  
    +**Types of changes**
    +
    +- *Changes in the number or type (i.e. different source) of input 
sources*: This is not allowed.
    +
    +- *Changes in the parameters of input sources*: Whether this is allowed 
and whether the semantics 
    +  of the change are well-defined depends on the source and the query. Here 
are a few examples.
    +
    +  - Addition/deletion/modification of rate limits is allowed: 
`spark.readStream.format("kafka").option("subscribe", "topic")` to 
`spark.readStream.format("kafka").option("subscribe", 
"topic").option("maxOffsetsPerTrigger", ...)`
    +
    +  - Changes to subscribed topics/files is generally not allowed as the 
results are unpredictable: 
`spark.readStream.format("kafka").option("subscribe", "topic")` to 
`spark.readStream.format("kafka").option("subscribe", "newTopic")`
    +
    +- *Changes in the type of output sink*: Changes between a few specific 
combinations of sinks 
    +  are allowed. This needs to be verified on a case-by-case basis. Here are 
a few examples.
    +
    +  - File sink to Kafka sink is allowed. Kafka will see only the new data.
    +
    +  - Kafka sink to file sink is not allowed.
    +
    +  - Kafka sink changed to foreach, or vice versa is allowed.
    +
    +- *Changes in the parameters of output sink*: Whether this is allowed and 
whether the semantics of 
    +  the change are well-defined depends on the sink and the query. Here are 
a few examples.
    +
    +  - Changes to output directory of a file sink is not allowed: 
`sdf.writeStream.format("parquet").option("path", "/somePath")` to 
`sdf.writeStream.format("parquet").option("path", "/anotherPath")`
    +
    +  - Changes to output topic is allowed: 
`sdf.writeStream.format("kafka").option("topic", "someTopic")` to 
`sdf.writeStream.format("kafka").option("path", "anotherTopic")`
    +
    +  - Changes to the user-defined foreach sink (that is, the `ForeachWriter` 
code) is allowed, but the semantics of the change depends on the code.
    +
    +- *Changes in projection / filter / map-like operations**: Some cases are 
allowed. For example:
    +
    +  - Addition / deletion of filters is allowed: `sdf.selectExpr("a")` to 
`sdf.where(...).selectExpr("a").filter(...)`.
    +
    +  - Changes in projections with same output schema is allowed: 
`sdf.selectExpr("stringColumn AS json").writeStream` to 
`sdf.select(to_json(...).as("json")).writeStream`.
    +
    +  - Changes in projections with different output schema are conditionally 
allowed: `sdf.selectExpr("a").writeStream` to `sdf.selectExpr("b").writeStream` 
is allowed only if the output sink allows the schema change from `"a"` to `"b"`.
    +
    +- *Changes in stateful operations*: Some operations in streaming queries 
need to maintain
    +  state data in order to continuously update the result. Structured 
Streaming automatically checkpoints
    +  the state data to fault-tolerant storage (for example, DBFS, AWS S3, 
Azure Blob storage) and restores it after restart.
    --- End diff --
    
    remove `DBFS`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to