xuanyuanking edited a comment on issue #23470: [SPARK-26549][PySpark] Fix for python worker reuse take no effect for Python3 URL: https://github.com/apache/spark/pull/23470#issuecomment-451857517 Thanks Wenchen Liangchi and Hyukjin for your comment, the JIRA description has more detailed and the code I added before: https://issues.apache.org/jira/browse/SPARK-26549. ~~I think the bug here triggered by Python2 has handled the -1 value while Python3 is not. The root cause is different behavior and call stack between Python2 and Python3, I'm still keeping tracking this, will give more detailed log and trace stack soon, any help and advise is appreciated.~~ Sorry for the mess, the bug only for `sc.parallelize(xrange(x))`, it's nothing to do with specific python version, I didn't realize that the code path difference between xrange and range in 'parallelize'... I'll change the JIRA and PR description.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
