dbtsai opened a new pull request, #142: URL: https://github.com/apache/spark-connect-rust/pull/142
### What changes were proposed in this pull request? Build `pyspark.sql.functions.__all__` using a private membership set, retaining the public list and its order. Snapshot globals before allocating temporary bookkeeping to preserve the existing `name`/`obj` attributes and reload behavior. Keep `builtin.__all__` a separate equal-order list. Add a fresh-process regression in the existing offline suite. ### Why are the changes needed? The export loop checks each callable name against a growing list. With 568 exports, that bookkeeping is quadratic. A temporary set makes membership checks expected constant time and the scan expected linear time. [SPARK-60742](https://issues.apache.org/jira/browse/SPARK-60742) ### Does this PR introduce _any_ user-facing change? No public API or behavior change. Export order, callable classes, aliases, signatures, star imports, synthetic submodules and legacy reload behavior are preserved. Module initialization performs less export-membership work. ### How was this patch tested? - Fresh-process export regression and existing Python offline/Arrow suites: 301 passed on each clean upstream role; 308 passed on each matched PR126 role; 1055 passed and 38 skipped on each frozen combined-v11 role. - Complete ordered export/public-global and callable-metadata comparisons on fresh import plus three reloads match for all three pairs. Export counts remain `[568, 568, 569, 569]`. Cloudpickle outcomes and 24 real mock-RPC serialized Plan captures match, including deterministic Plan bytes. Only unordered class metadata keys are canonicalized; identical-wheel repetition in 12 fresh processes independently reproduced descriptor-order variation. - 392 Rust client and 95 core unit tests, all native/WASM-feature integration test compilation, formatting, and four golden/serialization targets passed. Advisory Clippy exited 101 on unchanged Rust diagnostics. No JVM or live Spark server was used. - 1020 unique final performance samples passed: 960 samples across 28 first-use plan cases plus four warm guards per stack, three clients and five fresh samples per case/client; 60 import-only samples, fifteen per baseline/candidate role. All 320 paired Rust RPC counter maps match. Core cases record counters without expected-RPC assertions; these maps were separately compared. Byte counters are mock serialized-request lengths and Arrow fixture payload sizes. The upstream patch is based on `d9defae6` and excludes PR126. Performance minimal pairs apply PR126 equally; combined pairs use frozen published-only v11 (`856c08f3`) equally. Each Rust pair has identical native bytes and dependencies; source/wheel/installed-module provenance is checked before every timed session. The original frozen harness, timers and correctness validators are unchanged. Its official Python reference remains `5.0.0.dev0`; Rust wheels are `4.3.0`. Import-only timing starts after numpy, pandas, psutil, pyarrow and SparkSession prerequisites, before any session or plan: | Matched stack | Baseline import median | Candidate import median | |---|---:|---:| | minimal | 1.617 ms | 1.038 ms | | combined | 1.555 ms | 1.055 ms | All observed positive candidate median wall-time changes in the plan/warm matrix are disclosed below. For example, minimal warm collect increased by 0.047084 ms and linear expression by 0.09 microseconds per operation. All seven baseline/candidate sample ranges overlap. Five samples are exploratory; overlapping ranges are inconclusive and do not prove neutrality. These results do not establish a warmed-path speedup. | Stack | Case | Wall-time median increase ms/op | Wall-time median increase | |---|---|---:|---:| | minimal | `core-outbound-rpc-mixed-10` | 0.122166 | 5.01% | | minimal | `core-outbound-rpc-local-100000` | 0.271500 | 0.55% | | minimal | `api-select-width16` | 0.000001 | 0.16% | | minimal | `linear-expression-1` | 0.000090 | 2.35% | | minimal | `plan-chain-16-collect-empty` | 0.047084 | 9.16% | | combined | `chain-withColumn-16` | 0.000764 | 2.94% | | combined | `linear-expression-1` | 0.000064 | 1.77% | Median import RSS delta increased by 48 KiB in both matched stacks. This is resident-memory change captured immediately after import, before provenance checks; it does not measure allocated bytes. Full timing ranges retain the candidate construction joins5 sample at 14.388 ms and outbound deep10 sample at 11.669 ms. Cold import, first full-plan construction/outbound mock RPC, and warmed operations have separate measurement boundaries. RSS was sampled around plan measurements; import RSS was captured before provenance work. An early interrupted timing attempt was preserved and excluded; only the independently reviewed final control version contributes to the 1020 samples. Candidate CPU medians increased in 3 of 32 minimal cases and 2 of 32 combined cases. The largest increases were 0.306 ms/op for minimal outbound local100k and 0.001333 ms/op for combined chain-withColumn16. Absolute sampled-peak RSS medians increased in 16 minimal cases (maximum 1,622,016 bytes at construction local100k) and 19 combined cases (maximum 737,280 bytes at construction joins5). Other absolute RSS comparisons include pre-timer setup and process high-water effects. These measurements do not establish allocation reduction, neutrality, or a leak. Full time/CPU/RSS ranges and every pair metric are retained in `performance-all-cases.csv`, `performance-summary.json` and `adverse-costs.json`. | Stack | Case | CPU median increase ms/op | CPU median increase | |---|---|---:|---:| | minimal | `core-outbound-rpc-local-100000` | 0.306000 | 0.62% | | minimal | `linear-expression-1` | 0.000100 | 2.59% | | minimal | `plan-chain-16-collect-empty` | 0.021000 | 8.33% | | combined | `chain-withColumn-16` | 0.001333 | 5.00% | | combined | `linear-expression-1` | 0.000080 | 2.21% | Both median import RSS delta and median process-peak RSS increased by 48 KiB in each stack. Minimal import delta was 544→592 KiB and process peak 95,392→95,440 KiB; combined delta was 560→608 KiB and process peak 95,456→95,504 KiB. RSS was captured before provenance work; process peak includes prerequisite imports. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6). This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
