L. C. Hsieh created SPARK-60858:
-----------------------------------
Summary: Close remaining Spark 4.3.0 client API gaps in the Rust
drop-in
Key: SPARK-60858
URL: https://issues.apache.org/jira/browse/SPARK-60858
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: L. C. Hsieh
Fix For: connect-rust-4.3.0
Follow-up to SPARK-59305, which retargeted spark-connect-rust to Apache Spark
4.3.0. Close the remaining 4.3.0 client API gaps in the Rust-backed drop-in:
* Nanosecond timestamp types: map the TimestampNTZNanos / TimestampLTZNanos
proto kinds to their own DataTypes (instead of TIME / microsecond TIMESTAMP),
with proto, JSON, DDL and Arrow support and picklable Python types.
* functions: instr(occurrence=...) without start searches from position 1, and
None means omitted for instr / regexp_replace; variant_delete() with no paths
raises CANNOT_BE_EMPTY.
* SparkSession.__enter__ / __exit__ (exit stops the session); stop() is
idempotent and best-effort, and clears the active session and the cached
SQLContext.
* Spark Connect SQLContext / HiveContext (pyspark.sql.connect.context), with
SQLContext.getOrCreate() routing to it.
* Pipelines create_auto_cdc_flow: SCD type 2, track_history_column_list /
track_history_except_column_list and spark_conf, with the 4.3.0 client-side
validation.
* operation_id: ExecutionInfo.operation_id (DataFrame.executionInfo returns a
pyspark.sql.metrics.ExecutionInfo) and SparkConnectException.operation_id.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
