parthchandra commented on a change in pull request #24865: [SPARK-27100][SQL]
dag-scheduler-event-loop" java.lang.StackOverflowError
URL: https://github.com/apache/spark/pull/24865#discussion_r295556674
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
##########
@@ -735,4 +744,88 @@ abstract class BucketedReadSuite extends QueryTest with
SQLTestUtils {
df1.groupBy("j").agg(max("k")))
}
}
+
+ // a test with a single bucketed partition where the number of files in the
partition is large
+ // tests for the condition where the serialization of such a task may
result in a stack overflow
+ // if the files list is stored in a recursive data structure
+ test("SPARK-27100 stack overflow: read bucketed data with large partitions")
{
+ testLargeFilePartitionStackOverflow(true)
+ }
+
+ // a test with a single non-bucketed partition where the number of files in
the partition is
+ // large tests for the condition where the serialization of such a task may
result in a stack
+ // overflow if the files list is stored in a recursive data structure
+ test("SPARK-27100 stack overflow: read non bucketed data with large
partitions") {
+ testLargeFilePartitionStackOverflow(true)
Review comment:
Yes you're right. Careless mistake on my part. And as it turns out the unit
test was broken for the non-bucketed case. Fixed and unified the test cases
into one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
