Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/14803#discussion_r80556219
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -512,6 +512,12 @@ csvDF = spark \
These examples generate streaming DataFrames that are untyped, meaning
that the schema of the DataFrame is not checked at compile time, only checked
at runtime when the query is submitted. Some operations like `map`, `flatMap`,
etc. need the type to be known at compile time. To do those, you can convert
these untyped streaming DataFrames to typed streaming Datasets using the same
methods as static DataFrame. See the [SQL Programming
Guide](sql-programming-guide.html) for more details. Additionally, more details
on the supported streaming sources are discussed later in the document.
+### Schema inference and partition of streaming DataFrames/Datasets
+
+By default, Structured Streaming from file based sources requires you to
specify the schema, rather than rely on Spark to infer it automatically. This
restriction ensures a consistent schema will be used for the streaming query,
even in the case of failures. For ad-hoc use cases, you can reenable schema
inference by setting `spark.sql.streaming.schemaInference` to `true`.
--- End diff --
Schema inference can lead to many corner cases regarding if the inferred
schema is different after restart. So I think we should use a stronger language
that schema inference is not advisable in production uses.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
