Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/19498#discussion_r146514175
--- Diff: python/pyspark/streaming/util.py ---
@@ -64,7 +64,11 @@ def call(self, milliseconds, jrdds):
t = datetime.fromtimestamp(milliseconds / 1000.0)
r = self.func(t, *rdds)
if r:
- return r._jrdd
+ # Here, we work around to ensure `_jrdd` is `JavaRDD` by
wrapping it by `map`.
+ #
org.apache.spark.streaming.api.python.PythonTransformFunction requires to return
+ # `JavaRDD`; however, this could be `JavaPairRDD` by some
APIs, for example, `zip`.
+ # See SPARK-17756.
+ return r.map(lambda x: x)._jrdd
--- End diff --
wouldnt introducing this additional map cause a slight perf regression?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
