wayne-kyungwonpark commented on code in PR #33709:
URL: https://github.com/apache/spark/pull/33709#discussion_r1915951401
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetPartitionDiscoverySuite.scala:
##########
@@ -1067,7 +1067,7 @@ abstract class ParquetPartitionDiscoverySuite
test("SPARK-23436: invalid Dates should be inferred as String in partition
inference") {
withTempPath { path =>
- val data = Seq(("1", "2018-01", "2018-01-01-04", "test"))
+ val data = Seq(("1", "2018-41", "2018-01-01-04", "test"))
Review Comment:
@MaxGekk
Hello,
It's late, but I'd like to ask the 'my' test result related to this test
code.
In spark 3.3.0, I tested what is the read type of partition columns.
When the date_month column value was '2018-01-01', I was able to get the
`DateType` type, but I couldn't obtain it when the value was '2018-01' like
below.
```scala
val data = Seq(("1", "2018-01", "2018-01-01-04", "test"))
.toDF("id", "date_month", "date_hour", "data")
val path = "some_path"
data.write.partitionBy("date_month", "date_hour").parquet(path)
val input = spark.read.parquet(path).select("id", "date_month", "date_hour",
"data")
println(input.schema)
println(data.schema)
```
```
StructType(StructField(id,StringType,true),
StructField(date_month,StringType,true),
StructField(date_hour,StringType,true), StructField(data,StringType,true))
StructType(StructField(id,StringType,true),
StructField(date_month,StringType,true),
StructField(date_hour,StringType,true), StructField(data,StringType,true))
```
So, I would like to ask what it means to change the date_month value from
'2018-01' to '2018-41' in this test code.
Is there anything I might be misunderstanding or any mistakes in my testing?
Thanks.🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
