[ 
https://issues.apache.org/jira/browse/SPARK-61363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61363:
-----------------------------------
    Labels: pull-request-available  (was: )

> Match PySpark's literals, Observations, caches, nanosecond type names and 
> local iterators in the Python drop-in
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61363
>                 URL: https://issues.apache.org/jira/browse/SPARK-61363
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Low-priority findings of the s16 review of the Python drop-in, each checked 
> against official PySpark 4.3.0:
> - lit(), isin, when and sql args took an object that has only __index__ or 
> only __float__ (or a Fraction) as an int or a double; PySpark raises 
> UNSUPPORTED_DATA_TYPE.
> - DataFrame.transform did not assert that the function returned a DataFrame; 
> PySpark raises AssertionError "Func returned an instance of type [<class 
> 'int'>], should have been DataFrame."
> - Observation.get of a metric that PySpark's LiteralExpression._to_value does 
> not convert (a year-month interval, a nanosecond timestamp) returned an 
> internal marker; PySpark raises PySparkTypeError UNSUPPORTED_LITERAL from the 
> action, and then gives {} from get.
> - _parse_datatype_json_value, _parse_datatype_json_string and the fromJson of 
> StructType, StructField and ArrayType took the nanosecond names 
> timestamp_ntz(9) and timestamp_ltz(9); PySpark raises CANNOT_PARSE_DATATYPE. 
> TimestampNTZNanosType and TimestampLTZNanosType were attributes of 
> pyspark.sql.types, which PySpark does not have.
> - lit(pd.Timestamp("...123456789")) emitted pandas' UserWarning "Discarding 
> nonzero nanoseconds in conversion"; PySpark takes the microseconds silently.
> - A str that UTF-8 cannot encode (a lone surrogate) gave PySparkTypeError 
> "Unsupported DataType `str`" from lit and sql args; PySpark raises protobuf's 
> ValueError "... isn't a valid unicode string and can't be encoded in UTF-8."
> - toLocalIterator(prefetchPartitions=True) called the progress handlers on a 
> runtime thread ahead of the consumer, and an exception of a handler that the 
> consumer never reached stayed referenced for good; PySpark Connect ignores 
> prefetchPartitions and calls the handlers in next().
> - isStreaming, semanticHash, isEmpty and inputFiles asked the server at each 
> call; PySpark caches them per DataFrame (cached_property and 
> functools.cache), also after stop().
> - toArrow of invalid UTF-8 raises, where PySpark returns a table that is 
> invalid Arrow: recorded in dev/parity/known-differences.md rather than 
> reproduced.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to