Stephen0421 commented on code in PR #10569:
URL: https://github.com/apache/paimon/pull/10569#discussion_r4296634421
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/execution/PaimonStrategy.scala:
##########
@@ -474,11 +474,15 @@ case class LateralVectorSearchExec(
_.toPaimonDataField)).asJava)
}
val sparkRow = SparkInternalRow.create(resultRowType)
- val vectorSearchBuilder = innerTable
- .newBatchVectorSearchBuilder()
- .withVectorColumn(columnName)
- .withLimit(limit)
- .withOptions(options.asJava)
+ val vectorSearchBuilder = {
+ val builder =
+ if (new
CoreOptions(innerTable.options()).vectorSearchDistributeEnabled()) {
+ new SparkBatchVectorSearchBuilderImpl(innerTable)
Review Comment:
Thanks for catching this. I’ve updated the LATERAL execution path so that
distributed batch searches are submitted from the driver. The executor-local
path continues to use the regular batch reader.
The driver path now has explicit limits, with defaults of:
- 1,024 query vectors and topK ≤ 128.
- 64 KiB per query, including the complete outer row and the separate query
vector, checked on executors before collection.
- 64 MiB of retained joined-row bytes, checked row by row before copying.
Exceeding the budget stops consumption of the current batch and prevents
subsequent search batches.
These are retained-payload limits; the documentation explicitly excludes
temporary decoding, JVM object, and Spark transport overhead.
I also added tests using real Spark jobs for both primary-key and
data-evolution tables, plus coverage for:
- Input rejection before any search job is submitted.
- Input and result byte-limit boundaries.
- Stopping within an overflowing batch.
- Duplicate query vectors with topK > 1, including payload and score
validation.
All 11 tests in `DistributedLateralVectorSearchTest` pass locally with the
Spark 3 profile, and Spotless checks pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
