Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r196930368
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala
---
@@ -51,7 +51,7 @@ class ContinuousDataSourceRDD(
sc: SparkContext,
dataQueueSize: Int,
epochPollIntervalMs: Long,
- @transient private val readerFactories: Seq[InputPartition[UnsafeRow]])
+ private val readerFactories: Seq[InputPartition[UnsafeRow]])
--- End diff --
We need to be able to generate the full list of partitions from within a
single task in order for coalesce to work.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
