dongjoon-hyun commented on code in PR #52130:
URL: https://github.com/apache/spark/pull/52130#discussion_r2302202190
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala:
##########
@@ -264,6 +264,109 @@ private[sql] object ArrowConverters extends Logging {
}
}
+ /**
+ * This is a class that converts input data in the form of a Byte array to
InternalRow instances
+ * implementing the Iterator interface.
+ *
+ * The input data must be a valid Arrow IPC stream, this means that the
first message is always
+ * the schema followed by N record batches.
+ *
+ * @param input Input Data
+ * @param context Task Context for Spark
+ */
+ private[sql] class InternalRowIteratorFromIPCStream(
+ input: Array[Byte],
+ context: TaskContext) extends Iterator[InternalRow] {
+
+ // Keep all the resources we have opened in order, should be closed
+ // in reverse order finally.
+ private val resources = new ArrayBuffer[AutoCloseable]()
+
+ // Create an allocator used for all Arrow related memory.
+ protected val allocator: BufferAllocator =
ArrowUtils.rootAllocator.newChildAllocator(
+ s"to${this.getClass.getSimpleName}",
+ 0,
+ Long.MaxValue)
+ resources.append(allocator)
+
+ private val reader = try {
+ new ArrowStreamReader(new ByteArrayInputStream(input), allocator)
+ } catch {
+ case e: Exception =>
+ closeAll(resources.toSeq.reverse: _*)
+ throw new IllegalArgumentException(
+ s"Failed to create ArrowStreamReader: ${e.getMessage}", e)
+ }
+ resources.append(reader)
+
+ private val root: VectorSchemaRoot = try {
+ reader.getVectorSchemaRoot
+ } catch {
+ case e: Exception =>
+ closeAll(resources.toSeq.reverse: _*)
+ throw new IllegalArgumentException(
+ s"Failed to read schema from IPC stream: ${e.getMessage}", e)
Review Comment:
I'm not sure `Claude Code` add this new line or not. However, could you
revert this change in order to be consistent as much as possible with the
original patch of `master` branch (if there is no real reason), @grundprinzip ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
