Hyukjin Kwon created SPARK-61991:
------------------------------------
Summary: Check the session_id of the connection string as
PySpark's uuid.UUID(session_id, version=4), with its messages
Key: SPARK-61991
URL: https://issues.apache.org/jira/browse/SPARK-61991
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Problem
PySpark's ChannelBuilder.session_id (connect/client/core.py:438-447) checks the
session_id of
the connection string with Python's uuid.UUID(session_id, version=4). That
takes any text which,
with every "urn:" and "uuid:" removed, the braces stripped from both ends and
every "-" removed,
is 32 characters that int(hex, 16) reads, such as "uuid:<uuid>", "urn:<uuid>",
braces around the
32 digits, or hyphens anywhere. Its error is INVALID_SESSION_UUID_ID of the
text of the ValueError,
such as "badly formed hexadecimal UUID string". The session ID is then sent as
given, and the
server refuses those other forms as a handle at the first request.
The core (crates/spark-connect-core/src/channel.rs, ChannelBuilder::session_id)
checked it with
the uuid crate's Uuid::parse_str. SparkSession.builder.remote(...).create() and
the Rust API take
that path. It refused some of the forms that Python's uuid reads, with the
origin "invalid UUID".
spark.client's ChannelBuilder is Python's and was already right.
Repro (u = "550e8400-e29b-41d4-a716-446655440000")
SparkSession.builder.remote(f"sc://localhost:15002/;session_id=uuid:{u}").create()
SparkSession.builder.remote("sc://localhost:15002/;session_id={550e8400e29b41d4a716446655440000}").create()
SparkSession.builder.remote("sc://localhost:15002/;session_id=x").create()
Actual (drop-in):
- PySparkValueError [INVALID_SESSION_UUID_ID] Parameter value session_id must
be a valid UUID
format: invalid UUID, at create(), for all three.
Expected (pyspark-client 4.3.0):
- the first two create the session; its first request fails with the server's
INVALID_HANDLE.FORMAT;
- the third raises "... must be a valid UUID format: badly formed hexadecimal
UUID string".
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]