ManvithPanyam opened a new pull request, #39851: URL: https://github.com/apache/beam/pull/39851
### What does this PR do? Fixes `.xs()` on `DeferredDataFrame`/`DeferredSeries` when a key matches exactly one row and all index levels are selected. **The bug:** pandas reduces dimensionality on single-item `.xs()` matches (`DataFrame` → `Series`, `Series` → scalar), but Beam's implementation assumed a static output shape across partitions. Non-matching partitions return an empty container of the original type, so when the matching partition returned a dimensionality-reduced result, cross-partition `pd.concat` either raised `TypeError` (scalar concat) or silently produced a corrupted schema (NaN columns from a shape mismatch). **The fix:** when `key_size >= nlevels`, matching partitions are routed through a wrapped singleton stage that mirrors pandas' actual runtime output, then unwrapped to the real return type — instead of assuming the proxy shape holds at execution time. **Known limitation (documented in code):** the *proxy* schema (computed at graph-construction time from a 0-row template) can't know whether a key will match 1 row or several at runtime, so it always assumes the dimensionality-reduced type. If a key has duplicate matches, pandas returns the non-reduced container instead — this is fundamentally undecidable at proxy time, same class of limitation as `sort_values()`, `describe()`, and other data-dependent-shape operations already in this module. Tests exercising duplicate-match keys use `check_proxy=False` accordingly, with the reasoning documented inline. ### Fixes Fixes #28559 ### Tests Added regression coverage in `frames_test.py` for all reported failure modes: single-level index single match, MultiIndex 0-levels-remaining single match (both unique and duplicate-key datasets), and Series single-item `.xs()`. Full `frames_test.py` suite: 452 passed, 19 skipped, zero regressions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
