HyukjinKwon opened a new pull request, #257:
URL: https://github.com/apache/spark-connect-rust/pull/257
### What changes were proposed in this pull request?
- `join` and `lateralJoin` normalize `how` as official's `plan.Join` and
`plan.LateralJoin` do (lowercased, without underscores, `None` as inner) and
raise `AnalysisException` `UNSUPPORTED_JOIN_TYPE` for the others, when `join`
is called, as official does (`crates/pyspark-rs/src/dataframe.rs`).
`lateralJoin` accepts only inner, left outer and cross, as official. `on`
follows `plan.Join`: a list of Columns joins on their conjunction, which failed
before, and the other forms raise official's errors.
- `DataFrame.agg({})` raises `AssertionError: exprs should not be empty`;
dict aggregations raise official's errors for a non-str column
(`AttributeError`) or function (`AssertionError`), in a helper that
`DataFrame.agg` and `GroupedData.agg` share (`crates/pyspark-rs/src/group.rs`).
- `StreamingQuery.status` returns official's dict (`message`,
`isDataAvailable`, `isTriggerActive`), and `lastProgress` and `recentProgress`
return `StreamingQueryProgress` objects instead of JSON strings
(`crates/pyspark-rs/src/streaming.rs`).
### Why are the changes needed?
[SPARK-61066](https://issues.apache.org/jira/browse/SPARK-61066): against a
local Spark 4.3.0 server:
| Code | Current master | This change and PySpark 4.3.0 |
| --- | --- | --- |
| `df.join(df, "id", "foo")` | `ValueError: Invalid join type: foo` |
`AnalysisException [UNSUPPORTED_JOIN_TYPE]` |
| `df.join(df2, [a.x == b.y, a.x > -1])` | `TypeError` | joins on the
conjunction |
| `df.agg({})` | `[Row()]` | `AssertionError: exprs should not be empty` |
| `df.agg({"id": ["max"]})` | `TypeError` | `AssertionError` |
| `q.status["isDataAvailable"]` | `TypeError: ... not subscriptable` | a
bool |
| `q.lastProgress` | a JSON string | `StreamingQueryProgress` |
### Does this PR introduce _any_ user-facing change?
Yes, these APIs behave as in PySpark.
### How was this patch tested?
- New offline tests in `python/tests/test_dropin_offline.py` for the join
types sent and raised, the forms of `on`, and the dict aggregations, and live
tests in `python/tests/test_dropin_e2e.py` for an unsupported join type and the
status and progress of a streaming query.
- The offline suite (992 passed, 8 skipped) and the live join, aggregation
and streaming tests.
An `on` of a dict or tuple raises official's `AttributeError` at `join`,
where official raises it at `collect`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
