JingsongLi commented on code in PR #10569:
URL: https://github.com/apache/paimon/pull/10569#discussion_r4282338531
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/execution/PaimonStrategy.scala:
##########
@@ -474,11 +474,15 @@ case class LateralVectorSearchExec(
_.toPaimonDataField)).asJava)
}
val sparkRow = SparkInternalRow.create(resultRowType)
- val vectorSearchBuilder = innerTable
- .newBatchVectorSearchBuilder()
- .withVectorColumn(columnName)
- .withLimit(limit)
- .withOptions(options.asJava)
+ val vectorSearchBuilder = {
+ val builder =
+ if (new
CoreOptions(innerTable.options()).vectorSearchDistributeEnabled()) {
+ new SparkBatchVectorSearchBuilderImpl(innerTable)
Review Comment:
[P1] Do not launch Spark-dispatched batch reads inside LATERAL executor tasks
`createSearchContext()` and the later `readBatch()` run inside
`child.execute().mapPartitions`, so this selects the Spark-dispatched reader on
an executor. Once its split/bucket count reaches the distribution threshold, it
constructs `SparkEngineContext` and tries to start another Spark job using
`SparkSession.active()`. I reproduced this with a real local[2] Spark job and a
committed 32-row, 4-bucket PK table (`global-index.thread-num=1`): both
ordinary and distributed batch readers return `[[1], [32]]` on the driver; the
ordinary batch reader returns the same results inside an executor, but the
newly selected reader aborts the stage with `IllegalStateException: No active
or default Spark session found` from `SparkEngineContext.<init>`.
Data-evolution index/raw dispatch has the same executor-side dependency. Please
perform cluster dispatch from the driver, or retain local batch evaluation
within the existing LATERAL tasks. The new tests replace `mapInSpark` with a
Java stream, so
they cannot catch this failure; add a real executor-context regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
