Hyukjin Kwon created SPARK-61590:
------------------------------------
Summary: Raise PySpark's errors for bools and out-of-range ints in
bucketBy counts, window frame bounds and StorageLevel, in the Python client
Key: SPARK-61590
URL: https://issues.apache.org/jira/browse/SPARK-61590
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The Python client of spark-connect-rust took some int arguments with PyO3's
conversions, where PySpark 4.3.0's Connect client stores the value as given and
protobuf (upb, protobuf 7) converts it when the plan is built:
- DataFrameWriter.bucketBy(True, ...) sent 1 bucket, where PySpark raises
TypeError "Field spark.connect.WriteOperation.BucketBy.num_buckets: Expected an
int, got a boolean." at the write. A count of 0 or less (False too) sends no
BucketBy in PySpark, and a count beyond an int raises "Value out of range" at
the write, where ours raised OverflowError at bucketBy or sent the count.
- Window.rowsBetween/rangeBetween took True as 1, where PySpark raises the
TypeError of Literal.integer / Literal.long (False is the current row), and
raised "frame bound must be an int or FrameBound" for np.int64 and other types,
where PySpark raises the AssertionError of WindowFrame, or the TypeError of its
threshold comparison.
- StorageLevel validated its values at construction with PyO3 errors (1.5,
2**40), where PySpark's stores them as given and persist raises protobuf's
errors (a bool replication raises "Field
spark.connect.StorageLevel.replication: Expected an int, got a boolean.").
- sampleBy's seed check is shared with the other int fields (no behavior
change).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]