Hyukjin Kwon created SPARK-61363:
------------------------------------

             Summary: Match PySpark's literals, Observations, caches, 
nanosecond type names and local iterators in the Python drop-in
                 Key: SPARK-61363
                 URL: https://issues.apache.org/jira/browse/SPARK-61363
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Low-priority findings of the s16 review of the Python drop-in, each checked 
against official PySpark 4.3.0:
- lit(), isin, when and sql args took an object that has only __index__ or only 
__float__ (or a Fraction) as an int or a double; PySpark raises 
UNSUPPORTED_DATA_TYPE.
- DataFrame.transform did not assert that the function returned a DataFrame; 
PySpark raises AssertionError "Func returned an instance of type [<class 
'int'>], should have been DataFrame."
- Observation.get of a metric that PySpark's LiteralExpression._to_value does 
not convert (a year-month interval, a nanosecond timestamp) returned an 
internal marker; PySpark raises PySparkTypeError UNSUPPORTED_LITERAL from the 
action, and then gives {} from get.
- _parse_datatype_json_value, _parse_datatype_json_string and the fromJson of 
StructType, StructField and ArrayType took the nanosecond names 
timestamp_ntz(9) and timestamp_ltz(9); PySpark raises CANNOT_PARSE_DATATYPE. 
TimestampNTZNanosType and TimestampLTZNanosType were attributes of 
pyspark.sql.types, which PySpark does not have.
- lit(pd.Timestamp("...123456789")) emitted pandas' UserWarning "Discarding 
nonzero nanoseconds in conversion"; PySpark takes the microseconds silently.
- A str that UTF-8 cannot encode (a lone surrogate) gave PySparkTypeError 
"Unsupported DataType `str`" from lit and sql args; PySpark raises protobuf's 
ValueError "... isn't a valid unicode string and can't be encoded in UTF-8."
- toLocalIterator(prefetchPartitions=True) called the progress handlers on a 
runtime thread ahead of the consumer, and an exception of a handler that the 
consumer never reached stayed referenced for good; PySpark Connect ignores 
prefetchPartitions and calls the handlers in next().
- isStreaming, semanticHash, isEmpty and inputFiles asked the server at each 
call; PySpark caches them per DataFrame (cached_property and functools.cache), 
also after stop().
- toArrow of invalid UTF-8 raises, where PySpark returns a table that is 
invalid Arrow: recorded in dev/parity/known-differences.md rather than 
reproduced.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to