leaves12138 commented on code in PR #8442:
URL: https://github.com/apache/paimon/pull/8442#discussion_r3517895961
##########
docs/docs/pypaimon/multimodal-api.mdx:
##########
@@ -332,6 +332,115 @@ result = (
)
```
+## Row IDs
+
+Multimodal tables enable `row-tracking.enabled` by default, so each row has a
+Paimon system column named `_ROW_ID`. Use `_ROW_ID` as an internal coordination
+key for retrieval, reranking, inference, and training jobs. Keep user-visible
+document IDs, object keys, or primary keys in normal columns.
+
+Use `with_row_id()` to include `_ROW_ID` in scan or search results. The method
+appends `_ROW_ID` to the current projection; if no projection is set, it
returns
+all table columns plus `_ROW_ID`.
+
+```python
+candidates = (
+ docs.search([0.1, 0.2, 0.3], column="embedding")
+ .where("category = 'lake'")
+ .select(["id", "content"])
+ .limit(100)
+ .with_row_id()
+ .to_pandas()
+)
+
+row_ids = candidates["_ROW_ID"].tolist()
+```
+
+Use `take_row_ids` to fetch rows selected by an earlier scan, vector search,
+full-text search, hybrid search, sampler, or split manifest. Results are not
+guaranteed to follow the input row-id order. Include `_ROW_ID` and reorder on
+the client if order matters.
+
+```python
+payload = (
+ docs.take_row_ids(row_ids)
+ .select(["id", "content", "image"])
+ .with_row_id()
+ .to_arrow()
+ .to_pylist()
+)
+
+payload_by_row_id = {row["_ROW_ID"]: row for row in payload}
+ordered_payload = [payload_by_row_id[row_id] for row_id in row_ids]
+```
+
+This pattern keeps broad candidate generation cheap: first search or filter to
+produce a compact row-id manifest, then fetch only the columns needed by the
+next stage.
+
+```python
+# Stage 1: broad retrieval with a narrow projection.
+manifest = (
+ docs.search(query_vector, column="embedding")
+ .select(["id"])
+ .limit(500)
+ .with_row_id()
+ .to_pandas()
+)
+
+# Stage 2: expensive reranking payload, fetched only for candidates.
+rerank_payload = (
+ docs.take_row_ids(manifest["_ROW_ID"].tolist())
+ .select(["id", "title", "body"])
+ .with_row_id()
+ .to_list()
+)
+
+payload_by_row_id = {row["_ROW_ID"]: row for row in rerank_payload}
+rerank_inputs = [
+ {
+ "row_id": row_id,
+ "candidate_rank": rank,
+ "doc": payload_by_row_id[row_id],
+ }
+ for rank, row_id in enumerate(manifest["_ROW_ID"])
+]
+```
+
+For offline inference, feature backfills, or training splits, store row IDs in
a
+work queue or manifest table together with the source table snapshot or tag
used
Review Comment:
The docs recommend storing `_ROW_ID` with the source snapshot/tag, but
`take_row_ids` currently reads the latest table and the multimodal API has no
public way to pin this read to that snapshot/tag. In a manifest workflow, a row
id collected at snapshot N can disappear from `take_row_ids` after a delete or
overwrite, even though the docs say the caller should keep the snapshot/tag.
Could we add a time-travel shape for consuming row ids, for example
`take_row_ids(row_ids, snapshot_id=.../tag=...)`, `scan(snapshot_id=...)`, or
`table.at_snapshot(...)`, before documenting persisted manifests? Otherwise,
please narrow this section to short-lived latest-snapshot row-id usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
