This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 4482db27f5e8 [SPARK-60520][SS] Validate partition columns when
starting a file stream sink query
4482db27f5e8 is described below
commit 4482db27f5e8997818c249d8e29dda068db79480
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 24 12:31:03 2026 -0700
[SPARK-60520][SS] Validate partition columns when starting a file stream
sink query
### What changes were proposed in this pull request?
This PR aims to validate the partition columns of a file stream sink when a
streaming query starts, like batch file writes. `DataSource.createSink` now
receives the schema of the streaming query from `DataStreamWriter` and calls
`PartitioningUtils.validatePartitionColumn` before creating `FileStreamSink`,
as `DataSource.planForWritingFileFormat` does for batch writes.
This covers `DataStreamWriter.start` and `DataStreamWriter.toTable` with an
existing file source table. Spark Connect is also covered because its server
uses `DataStreamWriter`. Note that `DataStreamWriter.toTable` with a new table
already validates the partition columns when it creates the table.
In addition, this PR adds an entry to the Structured Streaming migration
guide because existing streaming queries with such partition columns fail to
restart.
### Why are the changes needed?
Apache Spark 2.0 validated the partition columns of a file stream sink in
`FileStreamSinkWriter`. However, SPARK-18025 removed `FileStreamSinkWriter` to
reuse `FileFormatWriter` in Apache Spark 2.1.0
([v2.1.0](https://github.com/apache/spark/releases/tag/v2.1.0), 2016-12-15),
and this validation has been missing since then. As a result, the file stream
sink accepts the partition columns which batch writes reject, and fails at
runtime or writes data which cannot be read back.
- `VARIANT` partition columns fail at the first micro-batch with a
compilation error of the generated code.
```
org.codehaus.commons.compiler.CompileException: File 'generated.java',
Line 20, Column 20: Failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 20,
Column 20: A method named "compare" is not declared in any enclosing class nor
any supertype, nor through a static import
```
- `ARRAY`, `MAP`, and `STRUCT` partition columns are written as directories
like `p=%5B1, 2%5D`, `p=%7B1 -> 2}`, and `p=%7B1}`, which cannot be read back
as the original values.
```scala
scala> spark.readStream.format("rate").load().selectExpr("value AS i",
"array(1, 2) AS
p").writeStream.format("parquet").partitionBy("p").option("checkpointLocation",
"/tmp/checkpoint").start("/tmp/output")
```
```
$ ls /tmp/output
_spark_metadata
p=%5B1, 2%5D
```
- Non-default collated string partition columns are partitioned by their
binary values. For example, `'a'` and `'A'` of `STRING COLLATE UTF8_LCASE` are
written as different partitions.
- Duplicate partition columns, e.g., `partitionBy("p", "p")`, are written
as nested directories like `p=0/p=0`, which fail to be read with
`COLUMN_ALREADY_EXISTS`.
- Partitioning by all columns fails at the first micro-batch for Parquet
and Text, and writes data which cannot be read back correctly for ORC
(`UNABLE_TO_INFER_SCHEMA`) and CSV (no rows).
### Does this PR introduce _any_ user-facing change?
Yes. A streaming query with a file stream sink fails at start with the same
errors as batch writes if its partition columns are invalid.
```
org.apache.spark.sql.AnalysisException:
[INVALID_PARTITION_COLUMN_DATA_TYPE] Cannot use "ARRAY<INT>" for partition
column. SQLSTATE: 0A000
```
```
org.apache.spark.sql.AnalysisException: [COLUMN_ALREADY_EXISTS] The column
`p` already exists. Choose another name or rename the existing column.
SQLSTATE: 42711
```
```
org.apache.spark.sql.AnalysisException: [ALL_PARTITION_COLUMNS_NOT_ALLOWED]
Cannot use all columns for partition columns. SQLSTATE: KD005
```
Note that existing streaming queries with such partition columns fail to
restart. This includes partitioning by all columns with JSON format, whose
output can be read back, but which batch writes have rejected since Apache
Spark 2.0.0 (SPARK-15743). The migration guide describes how to restart them.
### How was this patch tested?
Pass the CIs with the newly added test case in `FileStreamSinkSuite` and
the updated test case in `DataStreamReaderWriterSuite`, which fail without this
fix.
Previously, the test case `prevent all column partitioning` in
`DataStreamReaderWriterSuite` passed regardless of the validation, because
`spark.range(10).writeStream` fails before starting a query since
`spark.range(10)` is not a streaming `Dataset`. This PR updates it to use a
streaming `Dataset` and to check the error condition.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59759 from dongjoon-hyun/SPARK-60520.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit d93de8c0c244b01435295021974794a868fc2ad7)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/streaming/ss-migration-guide.md | 9 ++++++
.../spark/sql/classic/DataStreamWriter.scala | 2 +-
.../sql/execution/datasources/DataSource.scala | 6 ++--
.../spark/sql/streaming/FileStreamSinkSuite.scala | 35 ++++++++++++++++++++++
.../test/DataStreamReaderWriterSuite.scala | 18 ++++++-----
5 files changed, 60 insertions(+), 10 deletions(-)
diff --git a/docs/streaming/ss-migration-guide.md
b/docs/streaming/ss-migration-guide.md
index 3e9eb0e4448b..579ecaaf953b 100644
--- a/docs/streaming/ss-migration-guide.md
+++ b/docs/streaming/ss-migration-guide.md
@@ -27,6 +27,15 @@ Please refer [Migration Guide: SQL, Datasets and
DataFrame](../sql-migration-gui
- Since Spark 4.3, in Spark Connect the DataFrame passed to a Python
`foreachBatch` function is bound to the streaming query's own session, a clone
of the session that started the query, matching classic `foreachBatch`. This
runs the batch under the configuration the streaming engine pins on the clone
rather than the root session's. As a result, the batch DataFrame can no longer
be combined with the root session inside the function: a temporary view created
from it is not visible to the [...]
+- Since Spark 4.3, a streaming query with a file stream sink validates its
partition columns at
+ start like batch file writes. The query fails with
`INVALID_PARTITION_COLUMN_DATA_TYPE` if a
+ partition column has a type such as `ARRAY`, `MAP`, `STRUCT`, or a string
with a non-default
+ collation like `UTF8_LCASE`, with `COLUMN_ALREADY_EXISTS` if the partition
columns are
+ duplicated, and with `ALL_PARTITION_COLUMNS_NOT_ALLOWED` if all columns are
partition columns.
+ As a result, existing queries with such partition columns fail to restart.
For the first case,
+ change the type of the partition column without renaming it to keep the
directory layout. For
+ the other cases, change the partition columns and use a new checkpoint
location and output path.
+
## Upgrading from Structured Streaming 4.1 to 4.2
- Since Spark 4.2, restarting a streaming query from a checkpoint whose
metadata file is missing while the offset or commit logs contain data fails
with `STREAMING_CHECKPOINT_MISSING_METADATA_FILE`, instead of silently
generating a new query ID (which can duplicate data in exactly-once sinks).
Restore the metadata file or use a new checkpoint location. To restore the
previous behavior, set
`spark.sql.streaming.checkpoint.verifyMetadataExists.enabled` to `false`. (See
[SPARK-55058](https: [...]
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/classic/DataStreamWriter.scala
b/sql/core/src/main/scala/org/apache/spark/sql/classic/DataStreamWriter.scala
index 1301985e3015..62849aab56e1 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/classic/DataStreamWriter.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/classic/DataStreamWriter.scala
@@ -351,7 +351,7 @@ final class DataStreamWriter[T] private[sql](ds:
Dataset[T]) extends streaming.D
className = source,
options = optionsWithClusteringColumns,
partitionColumns = normalizedParCols.getOrElse(Nil))
- ds.createSink(outputMode)
+ ds.createSink(outputMode, this.ds.schema)
}
/** @inheritdoc */
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
index 5d6f51925553..3e3c00d738bc 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
@@ -336,8 +336,8 @@ case class DataSource(
}
}
- /** Returns a sink that can be used to continually write data. */
- def createSink(outputMode: OutputMode): Sink = {
+ /** Returns a sink that can be used to continually write data of the given
schema. */
+ def createSink(outputMode: OutputMode, schema: StructType): Sink = {
providingInstance() match {
case s: StreamSinkProvider =>
s.createSink(sparkSession.sqlContext, caseInsensitiveOptions,
partitionColumns, outputMode)
@@ -349,6 +349,8 @@ case class DataSource(
if (outputMode != OutputMode.Append) {
throw
QueryCompilationErrors.dataSourceOutputModeUnsupportedError(className,
outputMode)
}
+ PartitioningUtils.validatePartitionColumn(
+ schema, partitionColumns, conf.caseSensitiveAnalysis)
new FileStreamSink(sparkSession, path, fileFormat, partitionColumns,
caseInsensitiveOptions)
case _ =>
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala
index 87ec48c3743d..8bc5037f86c0 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala
@@ -744,6 +744,41 @@ abstract class FileStreamSinkSuite extends StreamTest {
}
assert(exception.getMessage.contains("is not absolute path."))
}
+
+ test("SPARK-60520: Validate partition columns when starting a query") {
+ val df = MemoryStream[Int].toDF()
+ withTempDir { dir =>
+ def startQuery(data: DataFrame, partitionColumns: String*):
StreamingQuery = {
+ data.writeStream
+ .option("checkpointLocation", new File(dir,
"checkpoint").getCanonicalPath)
+ .format("parquet")
+ .partitionBy(partitionColumns: _*)
+ .start(new File(dir, "output").getCanonicalPath)
+ }
+
+ Seq(
+ "parse_json('1')" -> "\"VARIANT\"",
+ "array(1, 2)" -> "\"ARRAY<INT>\"",
+ "map(1, 2)" -> "\"MAP<INT, INT>\"",
+ "named_struct('a', 1)" -> "\"STRUCT<a: INT>\"",
+ "'a' COLLATE UTF8_LCASE" -> "\"STRING COLLATE UTF8_LCASE\"").foreach {
+ case (expr, dataType) =>
+ checkError(
+ exception = intercept[AnalysisException] {
+ startQuery(df.selectExpr("value AS i", s"$expr AS p"), "p")
+ },
+ condition = "INVALID_PARTITION_COLUMN_DATA_TYPE",
+ parameters = Map("type" -> dataType))
+ }
+
+ checkError(
+ exception = intercept[AnalysisException] {
+ startQuery(df.selectExpr("value AS i", "value AS p"), "p", "p")
+ },
+ condition = "COLUMN_ALREADY_EXISTS",
+ parameters = Map("columnName" -> "`p`"))
+ }
+ }
}
object PendingCommitFilesTrackingManifestFileCommitProtocol {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala
index a1dfefe3fb89..55090b0c6c6c 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala
@@ -552,13 +552,17 @@ class DataStreamReaderWriterSuite extends StreamTest with
BeforeAndAfter {
test("prevent all column partitioning") {
withTempDir { dir =>
val path = dir.getCanonicalPath
- intercept[AnalysisException] {
- spark.range(10).writeStream
- .outputMode("append")
- .partitionBy("id")
- .format("parquet")
- .start(path)
- }
+ checkError(
+ exception = intercept[AnalysisException] {
+ MemoryStream[Long].toDS().toDF("id").writeStream
+ .option("checkpointLocation", newMetadataDir)
+ .outputMode("append")
+ .partitionBy("id")
+ .format("parquet")
+ .start(path)
+ },
+ condition = "ALL_PARTITION_COLUMNS_NOT_ALLOWED",
+ parameters = Map.empty)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]