juliuszsompolski commented on code in PR #41443:
URL: https://github.com/apache/spark/pull/41443#discussion_r1230140585
##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectStreamHandler.scala:
##########
@@ -174,12 +187,14 @@ object SparkConnectStreamHandler {
dataframe.queryExecution.executedPlan match {
case LocalTableScanExec(_, rows) =>
+ events.postFinished()
rowToArrowConverter(rows.iterator).foreach { case (bytes, count) =>
sendBatch(bytes, count)
}
case _ =>
SQLExecution.withNewExecutionId(dataframe.queryExecution,
Some("collectArrow")) {
val rows = dataframe.queryExecution.executedPlan.execute()
+ events.postFinished()
Review Comment:
The query didn't even start running at this point, so I don't think this
should be finished.
The query is only submitted for execution by the
`spark.sparkContext.submitJob` further down.
The query is finished when that future.onComplete happens.
In the meantime, the query asynchronously sends results back as soon as they
are available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
