uros-b commented on code in PR #58418:
URL: https://github.com/apache/spark/pull/58418#discussion_r3904636204
##########
python/pyspark/sql/tests/test_types.py:
##########
@@ -2215,6 +2217,86 @@ def test_daytime_interval_type(self):
for n, (a, e) in enumerate(zip(actual, expected)):
self.assertEqual(a, e, "%s does not match with %s" % (exprs[n],
expected[n]))
Review Comment:
No test drives a nanosecond value INTO a classic Python UDF (as an argument,
not a return). The collect-based assertions added at this head
(test_timestamp_nanos_type) now exercise the scalar toJava case
(TimestampNanosVal -> epochMicros) - the previously pickler-less silent-NULL
path, so that regression is covered; but the Python-UDF-input caller of toJava
remains unexercised. In particular the new nanosecond map-key-collision throw
branch, whose own comment cites "the Python UDF input path," is never reached:
test_timestamp_nanos_type_map_key_collision uses collect(), which trips the
earlier Python-side guard in classic/dataframe.py instead. A udf(lambda x: x,
TimestampNTZNanosType(9)) over a nanosecond column, plus a nanosecond-keyed-map
variant, would close the gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
