Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21560#discussion_r198052028
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala
---
@@ -51,7 +51,7 @@ class ContinuousDataSourceRDD(
sc: SparkContext,
dataQueueSize: Int,
epochPollIntervalMs: Long,
- @transient private val readerFactories: Seq[InputPartition[UnsafeRow]])
+ private val readerFactories: Seq[InputPartition[UnsafeRow]])
--- End diff --
Wait. I dont see the `readerFactories` object to be used anywhere other
than in getPartitions, where they are saved as part of
`ContinuousDataSourceRDDPartition` objects. And RDD.compute() seems to picking
it up from `ContinuousDataSourceRDDPartition` objects, and not from
`readerFactories`. So I dont think `readerFactories` needs to be serialized.
At the very least, rename `readerFactories` to `readerInputPartitions` for
consistency.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
