Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r198337615
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala
---
@@ -51,7 +51,7 @@ class ContinuousDataSourceRDD(
sc: SparkContext,
dataQueueSize: Int,
epochPollIntervalMs: Long,
- @transient private val readerFactories: Seq[InputPartition[UnsafeRow]])
+ private val readerFactories: Seq[InputPartition[UnsafeRow]])
--- End diff --
We list the partitions when computing the coalesce RDD. Should we instead
be packing the partitions into the partitions of the coalesce RDD? I'd assumed
it was valid to expect that rdd.partitions would work on executors, but maybe
it's not.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
