liferoad commented on code in PR #29399:
URL: https://github.com/apache/beam/pull/29399#discussion_r1392757914
##########
sdks/python/apache_beam/runners/worker/data_plane.py:
##########
@@ -510,7 +513,19 @@ def input_elements(
return
if self._exception:
raise self._exception from None
+ current_time = time.time()
+ if next_waiting_log_time <= current_time:
+ # If at the same time another instruction is waiting on input queue
+ # to become available, it is a sign of inefficiency in data plane.
+ _LOGGER.info(
+ 'Waiting to receive elements in input queue '
+ 'for instruction: %s for %.2f seconds.',
+ instruction_id,
+ current_time - start_time)
Review Comment:
can we add `log_interval_sec` in the log and mention this frequency?
Was this tested with Dataflow jobs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
