LuciferYang commented on a change in pull request #35262:
URL: https://github.com/apache/spark/pull/35262#discussion_r790517761
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
##########
@@ -147,12 +147,19 @@ class ParquetEncodingSuite extends
ParquetCompatibilityTest with SharedSparkSess
withTempPath { dir =>
val path = s"${dir.getCanonicalPath}/test.parquet"
- val data = (1 to 3).map { i =>
- ( i, i.toLong, i.toShort, Array[Byte](i.toByte), s"test_${i}",
- DateTimeUtils.fromJavaDate(Date.valueOf(s"2021-11-0" + i)),
- DateTimeUtils.fromJavaTimestamp(Timestamp.valueOf(s"2020-11-01
12:00:0" + i)),
- Period.of(1, i, 0), Duration.ofMillis(i * 100),
- new BigDecimal(java.lang.Long.toUnsignedString(i*100000))
+ // Have more than 2 * 4096 records (so we have multiple tasks and each
task
+ // reads at least twice from the reader). This will catch any issues
with state
+ // maintained by the reader(s)
+ // Add at least one string with a null
+ val data = (1 to 8197).map { i =>
+ ( i,
+ i.toLong, i.toShort, Array[Byte](i.toByte),
+ if (i % 2 == 1) s"test_${i}" else null,
Review comment:
test_$i
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
