Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/21559#discussion_r195516533
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala
---
@@ -294,6 +333,16 @@ class MemorySink(val schema: StructType, outputMode:
OutputMode) extends Sink
def clear(): Unit = synchronized {
batches.clear()
+ numRows = 0
+ }
+
+ private def truncateRowsIfNeeded(rows: Array[Row], maxRows: Int,
batchId: Long): Array[Row] = {
--- End diff --
nit: I'd document that maxRows is the remaining row capacity, not the
maximum row limit defined in the options
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
