nguyen1hc opened a new pull request, #56533:
URL: https://github.com/apache/spark/pull/56533
### What changes were proposed in this pull request?
This PR updates pandas-on-Spark `Series.str.findall` to support regex
patterns with multiple capture groups.
When the regex pattern has more than one capture group, pandas returns a
list of tuples. The existing pandas UDF return type expects an array of
strings, which can fail during Arrow conversion. This PR returns an array of
arrays of strings for this case and converts each tuple match into a list.
This PR also adds regression test coverage for `Series.str.findall` with
multiple capture groups.
### Why are the changes needed?
`Series.str.findall` currently cannot handle regex patterns that return
tuples, for example:
```python
ps.Series(["abc-123 def-456"]).str.findall("([a-z]+)-([0-9]+)")
```
This should return the captured groups instead of failing during conversion.
### Does this PR introduce _any_ user-facing change?
Yes.
Previously, `Series.str.findall` could fail for regex patterns with multiple
capture groups. With this change, it returns nested arrays representing the
captured groups.
### How was this patch tested?
Added a regression test in `SeriesStringOpsAdvTests.test_string_findall`.
Also ran:
```bash
python -m py_compile python/pyspark/pandas/strings.py
python/pyspark/pandas/tests/series/test_string_ops_adv.py
git diff --check
PYTHON_EXECUTABLE=python ./dev/lint-python --ruff
```
Note: the local `lint-python --ruff` wrapper skipped ruff because the `ruff`
command was not installed. The focused PySpark unittest could not start locally
because Spark jars have not been built in this checkout.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
