Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18732#discussion_r142577791
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowEvalPythonExec.scala
---
@@ -26,6 +26,25 @@ import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.execution.SparkPlan
import org.apache.spark.sql.types.StructType
+private class BatchIterator[T](iter: Iterator[T], batchSize: Int)
+ extends Iterator[Iterator[T]] {
+
+ override def hasNext: Boolean = iter.hasNext
+
+ override def next(): Iterator[T] = {
--- End diff --
The behavior of this `BatchIterator` seems to be a bit weird and seems to
not follow `Iterator`. For example, you can continue call its `next()` and
never consume it to the end.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
