Hyukjin Kwon created SPARK-61590:
------------------------------------

             Summary: Raise PySpark's errors for bools and out-of-range ints in 
bucketBy counts, window frame bounds and StorageLevel, in the Python client
                 Key: SPARK-61590
                 URL: https://issues.apache.org/jira/browse/SPARK-61590
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


The Python client of spark-connect-rust took some int arguments with PyO3's 
conversions, where PySpark 4.3.0's Connect client stores the value as given and 
protobuf (upb, protobuf 7) converts it when the plan is built:

- DataFrameWriter.bucketBy(True, ...) sent 1 bucket, where PySpark raises 
TypeError "Field spark.connect.WriteOperation.BucketBy.num_buckets: Expected an 
int, got a boolean." at the write. A count of 0 or less (False too) sends no 
BucketBy in PySpark, and a count beyond an int raises "Value out of range" at 
the write, where ours raised OverflowError at bucketBy or sent the count.
- Window.rowsBetween/rangeBetween took True as 1, where PySpark raises the 
TypeError of Literal.integer / Literal.long (False is the current row), and 
raised "frame bound must be an int or FrameBound" for np.int64 and other types, 
where PySpark raises the AssertionError of WindowFrame, or the TypeError of its 
threshold comparison.
- StorageLevel validated its values at construction with PyO3 errors (1.5, 
2**40), where PySpark's stores them as given and persist raises protobuf's 
errors (a bool replication raises "Field 
spark.connect.StorageLevel.replication: Expected an int, got a boolean.").
- sampleBy's seed check is shared with the other int fields (no behavior 
change).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to