Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19884#discussion_r157420989
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala
---
@@ -86,21 +86,16 @@ private[sql] object ArrowConverters {
val root = VectorSchemaRoot.create(arrowSchema, allocator)
val arrowWriter = ArrowWriter.create(root)
- var closed = false
-
context.addTaskCompletionListener { _ =>
- if (!closed) {
- root.close()
- allocator.close()
- }
+ root.close()
+ allocator.close()
--- End diff --
We can simplify 2 places in `ArrowPythonRunner` as well?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
