dtenedor commented on code in PR #42422:
URL: https://github.com/apache/spark/pull/42422#discussion_r1290471044
##########
python/pyspark/worker.py:
##########
@@ -550,7 +550,10 @@ def read_udtf(pickleSer, infile, eval_type):
# See `PythonUDTFRunner.PythonUDFWriterThread.writeCommand'
num_arg = read_int(infile)
- arg_offsets = [read_int(infile) for _ in range(num_arg)]
+ offsets = [read_int(infile) for _ in range(num_arg)]
+ names = [utf8_deserializer.loads(infile) if read_bool(infile) else None
for _ in range(num_arg)]
+ args_offsets = [offset for offset, name in zip(offsets, names) if name is
None]
+ kwargs_offsets = {name: offset for offset, name in zip(offsets, names) if
name is not None}
Review Comment:
here we zip the offsets and names and then filter the result twice. Can we
dedup it into one place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
