Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14803#discussion_r80120376
  
    --- Diff: docs/structured-streaming-programming-guide.md ---
    @@ -512,6 +512,10 @@ csvDF = spark \
     
     These examples generate streaming DataFrames that are untyped, meaning 
that the schema of the DataFrame is not checked at compile time, only checked 
at runtime when the query is submitted. Some operations like `map`, `flatMap`, 
etc. need the type to be known at compile time. To do those, you can convert 
these untyped streaming DataFrames to typed streaming Datasets using the same 
methods as static DataFrame. See the [SQL Programming 
Guide](sql-programming-guide.html) for more details. Additionally, more details 
on the supported streaming sources are discussed later in the document.
     
    +### Schema inference and partition of streaming DataFrames/Datasets
    +
    +You can specify the schema for streaming DataFrames/Datasets to create 
with the API as shown in above example (i.e., `userSchema`). Alternatively, for 
file-based streaming source, you can config it to infer the schema. By default, 
the configure of streaming schema inference 
`spark.sql.streaming.schemaInference` is turned off. If the streaming 
DataFrame/Dataset is partitioned, the partition columns will only be inferred 
if the partition directories are present when the stream starts. When schema 
inference is turned off, for all file-based streaming sources except for `text` 
format, you have to include partition columns in the user provided schema.
    --- End diff --
    
    By default, Structured Streaming from file based sources requires you to 
specify the schema, rather than relying on Spark to infer it automatically.  
This restriction ensures a consistent schema will be used for the streaming 
query, even in the case of failures.  For ad-hoc use cases, you can reenable 
schema inference by setting `spark.sql.streaming.schemaInference` to `true`.
    
    Partition discovery does occur for subdirectories that are named 
`/key=value/` and listing will automatically recurse into these directories.  
If these columns appear in the user provided schema, they will be filled in by 
Spark based on the path of the file being read.  The directories that make up 
the partitioning scheme must be present when the query starts and must remain 
static.  For example, it is okay to add `/data/year=2016/` when 
`/data/year=2015/` was present, but it is invalid to change the partitioning 
column (i.e. by creating the directory `/data/date=2016-04-17/`).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to