HyukjinKwon opened a new pull request, #81:
URL: https://github.com/apache/spark-connect-rust/pull/81
### What changes were proposed in this pull request?
Closes the remaining public **pyspark v4.2.0** API gaps in the Rust Spark
Connect client
drop-in, found by a rigorous introspection audit of `pyspark.sql.connect.*`
vs the
drop-in. Everything is mirrored from the official implementation and backed
by the Rust
core (with a thin PyO3/Python surface), and every addition has tests.
Highlights (per-commit):
- **functions**: higher-order functions
(`transform`/`filter`/`exists`/`forall`/`aggregate`/
`reduce`/`zip_with`/`transform_keys`/`transform_values`/`map_filter`/`map_zip_with`)
via a
faithful `_create_lambda`/`_invoke_higher_order_function` port, plus
`cume_dist`/`broadcast`/
`call_function`/`call_udf`/`column`/`random`;
`udtf`/`arrow_udtf`/`arrow_udf` decorators.
Fixed `CallFunction`/`call_function`/`call_udf` dropping their arguments.
- **Catalog**: the 10 DDL/metadata ops
(createDatabase/dropDatabase/dropTable/dropView/
truncateTable/analyzeTable/getCreateTableString/getTableProperties/listPartitions/listViews)
**and** the result classes
(Table/Database/Function/Column/CatalogMetadata/TablePartition),
so `listTables()` etc. return typed objects like reference pyspark.
- **types**: the object model on every type class
(json/jsonValue/typeName/simpleString/
needConversion/fromInternal/toInternal/fromDDL; StructType
toDDL/treeString/toNullable/
fromJson/fieldNames; StructField collation methods), and the **real class
hierarchy**
(DataType/AtomicType/NumericType/IntegralType/FractionalType/DatetimeType/AnyTimeType/
AnsiIntervalType/SpatialType) with the reference MRO so `isinstance(dt,
NumericType)` works.
New `GeometryType`/`GeographyType`, and the
`VariantVal`/`Geometry`/`Geography` value holders.
- **DataFrame/Column**: repartitionById, zipWithIndex, Column.transform,
DataFrameStatFunctions.sampleBy.
- **streaming/reader**: DataFrameReader/DataStreamReader.changes,
DataStreamReader.xml/name,
and a **native Rust `StreamingQueryManager` listener bus** (trait +
background dispatch) so
addListener/removeListener/close work (Python is a thin adapter).
- **UDF**: registerJavaFunction/registerJavaUDAF;
UserDefinedFunction.asNondeterministic;
UserDefinedTableFunction.asDeterministic.
- **misc**: RuntimeConfig.getAll; Row.count/index; PythonEvalType; TableArg
(df.asTable());
the UDTF analyze classes
(AnalyzeResult/AnalyzeArgument/PartitioningColumn/OrderingColumn/
SelectedColumn + SkipRestOfInputTableException);
`SparkSession.Builder.channelBuilder` now
reconstructs an sc:// URL from the builder instead of raising; re-export
paths
(`pyspark.StorageLevel`, `pyspark.sql.streaming.StreamingQueryListener`,
`pyspark.sql.{avro,protobuf}.functions`,
`pyspark.sql.functions.partitioning`).
- **Rust ergonomics**: every `Vec<Column>` API now takes
`impl IntoIterator<Item = impl Into<Column>>` (arrays, `vec!`, iterators,
and `&str` column
names), with `From<&str>/String for Column`. Docs/README/examples updated
to the array form.
After this PR the introspection audit shows **0 method gaps across all core
SQL classes**
(DataFrame/Column/SparkSession/GroupedData/Reader/Writer/Catalog/Window/Observation/streaming/
types) and **0 gaps in the functions module** vs pyspark v4.2.0.
### Why are the changes needed?
The drop-in under-exposed the (already near-complete) Rust core and diverged
on several
signatures; some newer 4.2.0 types/classes were missing entirely. This
brings the client to
method-for-method parity with pyspark v4.2.0's public Connect surface.
### Does this PR introduce _any_ user-facing change?
Yes — many previously-missing public APIs are now available; behavior
mirrors pyspark v4.2.0.
The only intentional non-additions are internal transport plumbing
(`Column.to_plan`,
`SparkSession.client`) that don't apply to a native Rust transport.
### How was this patch tested?
Rust golden/plan/unit tests + server-gated e2e tests (run against a live
Spark 4.2.0 Connect
server) + offline Python drop-in tests, all green; `cargo fmt`/no-stub audit
pass. Type
object-model output was verified byte-for-byte against official pyspark.
### Known follow-ups (not in this PR)
- `UserDefinedType` custom-UDT protocol (Python-subclassing extensibility;
core `DataType::Udt`
already exists).
- `StatefulProcessor` +
`DataFrame.transformWithState`/`transformWithStateInPandas` (a large new
streaming stateful engine).
- A Rust-native port of the variant binary codec
(`VariantVal.toJson/toPython/parseJson`
currently use the vendored, byte-identical upstream `variant_utils`).
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
