leaves12138 commented on PR #8666:
URL: https://github.com/apache/paimon/pull/8666#issuecomment-4982211510
I reproduced a blocking Spark regression introduced by the new primary-key
batch route.
For a primary-key table configured with `pk-vector.index.columns`, this
query shape fails:
```sql
SELECT q.id, r.id
FROM T AS q,
LATERAL (
SELECT id
FROM vector_search('T', 'embedding', q.embedding, 1)
) AS r
```
`PrimaryKeyBatchVectorRead` returns `PrimaryKeyVectorResult`, which uses
physical file positions. However, `LateralVectorSearchExec` still calls
`result.results()` as if every batch result used global row IDs
(`PaimonStrategy.scala:485`, `:527`, and `:541`). This reaches
`PrimaryKeyScoredResult.results()` and throws:
```text
UnsupportedOperationException: Primary-key scored results use physical file
positions, not global row ids.
```
Please add physical-position-aware Spark batch consumption (including
per-query scores and combined physical splits), or avoid routing this Spark
path to the primary-key batch reader until it is supported. Please also add
Spark regression coverage for both regular row projections and metadata-only
projections.
The focused Core suite passes 60/60; the issue is specifically at the Spark
batch-result consumption boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
