QuakeWang commented on code in PR #8465:
URL: https://github.com/apache/paimon/pull/8465#discussion_r3524465731
##########
paimon-python/pypaimon/ray/data_evolution_merge_join.py:
##########
@@ -584,6 +584,139 @@ def _apply_group(group: pa.Table) -> pa.Table:
return all_msgs, num_updated, action_row_ids
+def _read_output_schema(table, read_cols: Sequence[str]) -> "pa.Schema":
+ """Result schema: each projected column's type plus int64 ``_ROW_ID``, in
+ ``read_cols`` order. Shared by the empty-result paths so they can't
drift."""
+ from pypaimon.schema.data_types import PyarrowFieldParser
+ from pypaimon.table.special_fields import SpecialFields
+
+ rid = SpecialFields.ROW_ID.name
+ full = PyarrowFieldParser.from_paimon_schema(table.table_schema.fields)
+ return pa.schema([
+ (col, pa.int64() if col == rid else full.field(col).type)
Review Comment:
This schema construction drops field nullability because `(name, type)`
defaults to nullable fields. That makes empty `read_by_row_id` results expose a
different schema from non-empty reads, especially for `_ROW_ID`, which should
be non-nullable per `SpecialFields.ROW_ID`.
Could we preserve `full.field(col)` for table columns and use `pa.field(rid,
pa.int64(), nullable=False)` for `_ROW_ID`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
