HyukjinKwon commented on a change in pull request #29023:
URL: https://github.com/apache/spark/pull/29023#discussion_r451220383
##########
File path: python/pyspark/rdd.py
##########
@@ -141,7 +141,9 @@ def _parse_memory(s):
def _load_from_socket(sock_info, serializer):
- (sockfile, sock) = local_connect_and_auth(*sock_info)
+ port = sock_info[0]
+ auth_secret = sock_info[1]
+ (sockfile, sock) = local_connect_and_auth(port, auth_secret)
Review comment:
This was correctly ported back at
https://github.com/apache/spark/pull/25593. There's no such code you pointed
out in Spark 2.4.4:
https://github.com/apache/spark/blob/v2.4.4/python/pyspark/sql/dataframe.py#L2182.
Are you using your own fork or mixing the Spark versions? Your error message
seems from
https://github.com/apache/spark/blob/v2.4.3/python/pyspark/sql/dataframe.py#L2179
which is Spark 2.4.3. Spark 2.4.3 does not have this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
