JingsongLi commented on code in PR #8442:
URL: https://github.com/apache/paimon/pull/8442#discussion_r3517970567
##########
docs/docs/pypaimon/multimodal-api.mdx:
##########
@@ -332,6 +332,115 @@ result = (
)
```
+## Row IDs
+
+Multimodal tables enable `row-tracking.enabled` by default, so each row has a
+Paimon system column named `_ROW_ID`. Use `_ROW_ID` as an internal coordination
+key for retrieval, reranking, inference, and training jobs. Keep user-visible
+document IDs, object keys, or primary keys in normal columns.
+
+Use `with_row_id()` to include `_ROW_ID` in scan or search results. The method
+appends `_ROW_ID` to the current projection; if no projection is set, it
returns
+all table columns plus `_ROW_ID`.
+
+```python
+candidates = (
+ docs.search([0.1, 0.2, 0.3], column="embedding")
+ .where("category = 'lake'")
+ .select(["id", "content"])
+ .limit(100)
+ .with_row_id()
+ .to_pandas()
+)
+
+row_ids = candidates["_ROW_ID"].tolist()
+```
+
+Use `take_row_ids` to fetch rows selected by an earlier scan, vector search,
+full-text search, hybrid search, sampler, or split manifest. Results are not
+guaranteed to follow the input row-id order. Include `_ROW_ID` and reorder on
+the client if order matters.
+
+```python
+payload = (
+ docs.take_row_ids(row_ids)
+ .select(["id", "content", "image"])
+ .with_row_id()
+ .to_arrow()
+ .to_pylist()
+)
+
+payload_by_row_id = {row["_ROW_ID"]: row for row in payload}
+ordered_payload = [payload_by_row_id[row_id] for row_id in row_ids]
+```
+
+This pattern keeps broad candidate generation cheap: first search or filter to
+produce a compact row-id manifest, then fetch only the columns needed by the
+next stage.
+
+```python
+# Stage 1: broad retrieval with a narrow projection.
+manifest = (
+ docs.search(query_vector, column="embedding")
+ .select(["id"])
+ .limit(500)
+ .with_row_id()
+ .to_pandas()
+)
+
+# Stage 2: expensive reranking payload, fetched only for candidates.
+rerank_payload = (
+ docs.take_row_ids(manifest["_ROW_ID"].tolist())
+ .select(["id", "title", "body"])
+ .with_row_id()
+ .to_list()
+)
+
+payload_by_row_id = {row["_ROW_ID"]: row for row in rerank_payload}
+rerank_inputs = [
+ {
+ "row_id": row_id,
+ "candidate_rank": rank,
+ "doc": payload_by_row_id[row_id],
+ }
+ for rank, row_id in enumerate(manifest["_ROW_ID"])
+]
+```
+
+For offline inference, feature backfills, or training splits, store row IDs in
a
+work queue or manifest table together with the source table snapshot or tag
used
Review Comment:
Thanks, this was a good catch. I added the time-travel shape instead of
narrowing the docs: `scan`, `search`, `search_vectors`, `search_hybrid`, and
`take_row_ids` now accept mutually-exclusive `snapshot_id` / `tag_name`
arguments. `take_row_ids` consumes row-id manifests against the pinned table
copy, and the docs now show passing the stored source snapshot when consuming
persisted manifests. I also added a regression test where the latest table
deletes the row but `take_row_ids(..., snapshot_id=...)` and `take_row_ids(...,
tag_name=...)` still fetch it from the pinned source snapshot/tag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
