wangzhigang1999 opened a new pull request, #11169: URL: https://github.com/apache/paimon/pull/11169
### Purpose Closes #11168. For a column-list projection such as `['value', 'rid']`, the reader can already return the requested columns in order. The extra output projection then recomputes the adapter and schema and wraps the same arrays in another `RecordBatch` for each batch. Skip this step in `TableRead.__init__` only when: - The projection is unnamed and its output names and source paths exactly match the top-level fields in `read_type`, in the same order. - The reader adapter returns those same fields and paths, without flattening nested fields or changing their types. Reuse the computed adapter when projection is still needed. This also covers directly constructed `TableRead` instances. Keep the builder's projection so `explain()` continues to show the user's request. ### Tests - Targeted tests passed with Python and Native execution, covering direct construction, nested fields, aliases, duplicate columns, empty projections, filters on columns omitted from the output, authorization and `explain()`. - Ruff and ty reported no new diagnostics in the changed files compared with the baseline. Compared with `6d50c1ab` on a telemetry table with 1,151,842 rows and TPC-H SF10 tables. Each query ran three times per version after a warmup, alternating between versions. The table shows median elapsed time; negative change means faster. | Query | Before (s) | After (s) | Change | | --- | ---: | ---: | ---: | | Telemetry: all columns | 5.577 | 5.604 | +0.5% | | Telemetry: one column | 0.345 | 0.311 | -9.9% | | Telemetry: reordered columns | 0.477 | 0.446 | -6.4% | | Telemetry: full MAP column | 5.654 | 5.553 | -1.8% | | Telemetry: filter with no matches | 0.252 | 0.251 | -0.4% | | Telemetry: filter on an omitted column | 0.248 | 0.247 | -0.1% | | Telemetry: batch iteration | 0.478 | 0.447 | -6.6% | | SF10: orders narrow projection | 0.636 | 0.521 | -18.2% | | SF10: Q6 scan and filter | 7.425 | 6.771 | -8.8% | | SF10: customer segment filter | 0.348 | 0.330 | -5.2% | | SF10: part 10% range filter | 0.231 | 0.228 | -1.7% | For every run, returned rows and schema matched the expected results computed from the source data. These measurements used the Python reader with Python 3.11 and Arrow 19, local append-only Parquet tables, separate processes and warm OS page cache. Timing includes reading and collecting results, plus Arrow IPC writes for TPC-H. Q6 measures its scan and filter, not the full SQL query. Native, Ray and OSS performance were not measured. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
