Yicong-Huang commented on code in PR #59246:
URL: https://github.com/apache/spark/pull/59246#discussion_r4192029568
##########
python/pyspark/eval_handlers/_pandas.py:
##########
@@ -75,22 +92,24 @@ def run(self, split_index: int, data:
"Iterator[pa.RecordBatch]") -> "Iterator[p
num_rows = input_batch.num_rows
# Input: Arrow -> pandas Series (struct columns become DataFrames).
- pandas_columns = ArrowToPandasConversion.to_pandas(
- input_batch,
- timezone=runner_conf.timezone,
- struct_in_pandas="dict",
- ndarray_as_list=False,
- prefer_int_ext_dtype=runner_conf.prefer_int_ext_dtype,
- df_for_struct=True,
- )
+ with self._input_timer:
+ pandas_columns = ArrowToPandasConversion.to_pandas(
+ input_batch,
+ timezone=runner_conf.timezone,
+ struct_in_pandas="dict",
+ ndarray_as_list=False,
+ prefer_int_ext_dtype=runner_conf.prefer_int_ext_dtype,
+ df_for_struct=True,
+ )
# Process: evaluate each UDF column-wise on pandas Series.
results = []
for udf_func, args_offsets, kwargs_offsets, return_type in
self._udfs:
- result = udf_func(
- *[pandas_columns[o] for o in args_offsets],
- **{k: pandas_columns[v] for k, v in
kwargs_offsets.items()},
- )
+ # Argument selection and return validation are outside the
callback timer.
+ args = [pandas_columns[o] for o in args_offsets]
+ kwargs = {k: pandas_columns[v] for k, v in
kwargs_offsets.items()}
Review Comment:
technically, the time spent on preparing args for udf should also be counted
for input time right?
the idea is that we want to make input time + udf time + output time = total
wall time of a worker.
but current way of instrumentation, some of the code are not counted for any
of those times, for example those highlighted lines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
