Hyukjin Kwon created SPARK-61367:
------------------------------------
Summary: Match PySpark and the server in the Rust API's pivot
literals, year-month intervals, range repartitions, sample_by strata, DDL,
Arrow batch checks, release_session, listener panics and handler threads
Key: SPARK-61367
URL: https://issues.apache.org/jira/browse/SPARK-61367
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Low-priority findings of the s16 review of the Rust API (apache-spark-connect),
each checked against official PySpark 4.3.0, the server or the Scala client:
- pivot sent a Float value as a double literal, so its column was named
"1.100000023841858" where the pivot without values (and the Scala client) names
it "1.1"; Byte and Short were sent as ints; and its NOT_EXPECTED_TYPE message
named only bool, float, int and str.
- Value had no year-month interval: collect gave Value::String("1-0"), so a
pivot on collected values named the column "1-0", not "INTERVAL '1-0' YEAR TO
MONTH".
- repartition_by_range with no columns was split round-robin by the server;
PySpark raises CANNOT_BE_EMPTY (and VALUE_NOT_POSITIVE for a count that is not
positive). sample_by turned a stratum that is not a literal into an unset one;
PySpark raises DISALLOWED_TYPE_FOR_CONTAINER.
- DataType::from_ddl rejected whitespace before "<" (array <int>, STRUCT
<a:int>), which the server accepts.
- PySpark checks the start_offset and the row count of each Arrow batch of a
result; ours did not, and grew the joined buffer of chunked batches without
reserving it.
- SparkSession::release_session after stop() or close() still sent
ReleaseSession; PySpark raises NO_ACTIVE_SESSION.
- A native streaming listener's panic on an asynchronous event failed the whole
subscription and cleared every listener; PySpark's post_to_all isolates each
listener.
- In a current_thread Tokio runtime, the progress handlers' update calls ran on
a helper thread; PySpark calls them on the thread that iterates the responses.
- With several invalid UTF-8 values in one struct or map column, ours reported
the first row by row; PySpark reports field by field, keys before values.
- A Raw relation (a plan of pyspark.ml) counted as one level for the depth
bound whatever its nesting, and cloning one at the bound overflowed a small
stack; LogicalPlan::to_proto has no bounded variant.
- Docs: the E0509 of moving fields out of a LogicalPlan (it implements Drop),
and that is_local() is a client-side guess.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]