Akash3121 commented on code in PR #10164:
URL: https://github.com/apache/paimon/pull/10164#discussion_r4096909149
##########
paimon-python/pypaimon/table/system/system_table_read.py:
##########
@@ -95,12 +96,20 @@ def to_duckdb(
return con
def _materialise(self, splits: List[Split]) -> pyarrow.Table:
- if self.predicate is not None:
+ from pypaimon.read.push_down_utils import
predicate_supports_arrow_filter
+
+ if self.predicate is not None and not predicate_supports_arrow_filter(
Review Comment:
Please preserve the existing rejection path for unsupported predicate
objects before calling this helper.
`SystemReadPipelineTest.test_filter_is_not_supported_yet` passes `object()`
and expects the public `NotImplementedError` , but
`predicate_supports_arrow_filter` immediately dereferences `.method` , so
this PR changes the result to an internal `AttributeError` . The focused
system-table suite currently fails on this test. An
`isinstance(self.predicate, Predicate)` guard (or making the helper return
`False` for unknown objects) would retain the previous error contract.
##########
paimon-python/pypaimon/table/system/system_table_read.py:
##########
@@ -110,6 +119,10 @@ def _materialise(self, splits: List[Split]) ->
pyarrow.Table:
+ type(split).__name__
)
arrow_table = split.arrow_table()
+ # Filter on the full schema before projection so a predicate may
+ # reference columns that are not part of the projection.
+ if arrow_predicate is not None:
Review Comment:
`predicate_supports_arrow_filter` only checks the predicate method, so it
accepts comparisons for every field type. System tables expose array fields
such as `$files.write_cols` and `$table_indexes.dv_ranges` , but PyArrow has
no equality kernel for list values. For example, `builder.equal("write_cols",
["a"])` reaches this line and raises `ArrowNotImplementedError: Function
'equal' has no kernel matching input types (list<item: string>, list<item:
string>)` . Please make the support check field-type/method aware (or provide
the existing row-wise fallback) and reject unsupported combinations with the
documented `NotImplementedError` . A regression test using an array-valued
system-table column would cover this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
