HyukjinKwon commented on code in PR #59315:
URL: https://github.com/apache/spark/pull/59315#discussion_r4237420971
##########
sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala:
##########
@@ -2877,6 +2896,28 @@ class BasicCharVarcharTestSuite extends
SharedSparkSession {
}
}
+ test("SPARK-60102: CSV CHAR filters are not pushed down") {
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ withTempPath { path =>
+ Seq("ab,1",
"xy,2").toDS().repartition(1).write.text(path.getCanonicalPath)
+ val df = spark.read.schema("c CHAR(4), i
INT").csv(path.getCanonicalPath)
+ checkAnswer(df.selectExpr("concat('<', c, '>')", "i"),
+ Seq(Row("<ab >", 1), Row("<xy >", 2)))
+ df.createOrReplaceTempView("csv_char")
Review Comment:
**Nit (P3):** `csv_char` is registered on the shared session and never
dropped. `BasicCharVarcharTestSuite` shares one `SparkSession`, and `afterEach`
only clears the cache, so the view outlives this test. Once `withTempPath`
exits, it also points at a deleted directory. Nothing collides today, but
wrapping this block in `withTempView("csv_char") { ... }`, or filtering `df`
directly, keeps the test self-contained.
**Verification:**
- **Behavior:** The SPARK-60102 CSV CHAR test still passes with unchanged
assertions.
- **Inspection:** The csv_char registration sits inside a scope that drops
the view on exit, or the test no longer registers a view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
