[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[ https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325756=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325756 ] ASF GitHub Bot logged work on BEAM-8333: Author: ASF GitHub Bot Created on: 09/Oct/19 15:26 Start Date: 09/Oct/19 15:26 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #9746: [BEAM-8333] Adding lull logging for SDK harness URL: https://github.com/apache/beam/pull/9746 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 325756) Time Spent: 2h 20m (was: 2h 10m) > Python SDK Worker should log lulls with progress-reporting thread > - > > Key: BEAM-8333 > URL: https://issues.apache.org/jira/browse/BEAM-8333 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[ https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325421=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325421 ] ASF GitHub Bot logged work on BEAM-8333: Author: ASF GitHub Bot Created on: 09/Oct/19 01:25 Start Date: 09/Oct/19 01:25 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9746: [BEAM-8333] Adding lull logging for SDK harness URL: https://github.com/apache/beam/pull/9746#discussion_r332796781 ## File path: sdks/python/apache_beam/runners/portability/fn_api_runner_test.py ## @@ -21,7 +21,9 @@ import logging import os import random +import re Review comment: You can remove this import now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 325421) Time Spent: 1h 50m (was: 1h 40m) > Python SDK Worker should log lulls with progress-reporting thread > - > > Key: BEAM-8333 > URL: https://issues.apache.org/jira/browse/BEAM-8333 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[
https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325423=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325423
]
ASF GitHub Bot logged work on BEAM-8333:
Author: ASF GitHub Bot
Created on: 09/Oct/19 01:25
Start Date: 09/Oct/19 01:25
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #9746: [BEAM-8333]
Adding lull logging for SDK harness
URL: https://github.com/apache/beam/pull/9746#discussion_r332796540
##
File path: sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
##
@@ -1580,6 +1584,38 @@ def test_split_half(self):
raise unittest.SkipTest("This test is for a single worker only.")
+class FnApiBasedLullLoggingTest(unittest.TestCase):
+ def create_pipeline(self):
+return beam.Pipeline(
+runner=fn_api_runner.FnApiRunner(
+default_environment=beam_runner_api_pb2.Environment(
+urn=python_urns.EMBEDDED_PYTHON_GRPC),
+progress_request_frequency=0.5))
+
+ def test_lull_logging(self):
+
+if sys.version_info < (3, 4):
Review comment:
Ack. You could add a BEAM-1251 todo here so that we can remove it after
python 2 support is removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325423)
Time Spent: 2h 10m (was: 2h)
> Python SDK Worker should log lulls with progress-reporting thread
> -
>
> Key: BEAM-8333
> URL: https://issues.apache.org/jira/browse/BEAM-8333
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-harness
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[
https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325419=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325419
]
ASF GitHub Bot logged work on BEAM-8333:
Author: ASF GitHub Bot
Created on: 09/Oct/19 01:22
Start Date: 09/Oct/19 01:22
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #9746: [BEAM-8333]
Adding lull logging for SDK harness
URL: https://github.com/apache/beam/pull/9746#discussion_r332796401
##
File path: sdks/python/apache_beam/runners/worker/sdk_worker.py
##
@@ -403,10 +412,35 @@ def process_bundle_split(self, request, instruction_id):
instruction_id=instruction_id,
error='Instruction not running: %s' % instruction_id)
+ def _log_lull_in_bundle_processor(self, processor):
+state_sampler = processor.state_sampler
+sampler_info = state_sampler.get_info()
+if (sampler_info
+and sampler_info.time_since_transition
+and sampler_info.time_since_transition > self.log_lull_timeout_ns):
+ step_name = sampler_info.state_name.step_name
+ state_name = sampler_info.state_name.name
+ state_lull_log = (
+ 'There has been a processing lull of over %.2f seconds in state %s'
+ % (sampler_info.time_since_transition / 1e9, state_name))
+ step_name_log = (' in step %s ' % step_name) if step_name else ''
+
+ exec_thread = getattr(sampler_info, 'tracked_thread', None)
+ if exec_thread is not None:
+thread_frame = sys._current_frames().get(exec_thread.ident) # pylint:
disable=protected-access
+stack_trace = '\n'.join(
+traceback.format_stack(thread_frame)) if thread_frame else ''
+ else:
+stack_trace = '-NOT AVAILABLE-'
+
+ logging.warning(
+ '%s%s. Traceback:\n%s', state_lull_log, step_name_log, stack_trace)
+
def process_bundle_progress(self, request, instruction_id):
# It is an error to get progress for a not-in-flight bundle.
-processor = self.bundle_processor_cache.lookup(
-request.instruction_id)
+processor = self.bundle_processor_cache.lookup(request.instruction_id)
+if processor:
+ self._log_lull_in_bundle_processor(processor)
Review comment:
I have not benchmarked this, but a similar operation is done on the python
prod worker, and it has not been a problem. Furthermore, a progress update
involves going through all of the user and system metrics, and building updates
for them. This should be equivalent to adding one more metric, as speaking to
the state sampler is lockless.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325419)
Time Spent: 1h 40m (was: 1.5h)
> Python SDK Worker should log lulls with progress-reporting thread
> -
>
> Key: BEAM-8333
> URL: https://issues.apache.org/jira/browse/BEAM-8333
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-harness
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[
https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325417=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325417
]
ASF GitHub Bot logged work on BEAM-8333:
Author: ASF GitHub Bot
Created on: 09/Oct/19 01:21
Start Date: 09/Oct/19 01:21
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #9746: [BEAM-8333]
Adding lull logging for SDK harness
URL: https://github.com/apache/beam/pull/9746#discussion_r332796139
##
File path: sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
##
@@ -1580,6 +1584,38 @@ def test_split_half(self):
raise unittest.SkipTest("This test is for a single worker only.")
+class FnApiBasedLullLoggingTest(unittest.TestCase):
+ def create_pipeline(self):
+return beam.Pipeline(
+runner=fn_api_runner.FnApiRunner(
+default_environment=beam_runner_api_pb2.Environment(
+urn=python_urns.EMBEDDED_PYTHON_GRPC),
+progress_request_frequency=0.5))
+
+ def test_lull_logging(self):
+
+if sys.version_info < (3, 4):
+ self.skipTest('Log-based assertions are supported after Python 3.4')
+try:
+ utils.check_compiled('apache_beam.runners.worker.opcounters')
+except RuntimeError:
+ self.skipTest('Cython is not available')
+
+with self.assertLogs(level='WARNING') as logs:
+ with self.create_pipeline() as p:
+sdk_worker.DEFAULT_LOG_LULL_TIMEOUT_NS = 1000 * 1000 # Lull after 1 ms
+
+_ = (p
+ | beam.Create([1])
+ | beam.Map(time.sleep))
+
+self.assertTrue(
Review comment:
Done!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325417)
Time Spent: 1h 20m (was: 1h 10m)
> Python SDK Worker should log lulls with progress-reporting thread
> -
>
> Key: BEAM-8333
> URL: https://issues.apache.org/jira/browse/BEAM-8333
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-harness
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[
https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325416=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325416
]
ASF GitHub Bot logged work on BEAM-8333:
Author: ASF GitHub Bot
Created on: 09/Oct/19 01:20
Start Date: 09/Oct/19 01:20
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #9746: [BEAM-8333]
Adding lull logging for SDK harness
URL: https://github.com/apache/beam/pull/9746#discussion_r332796127
##
File path: sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
##
@@ -1580,6 +1584,38 @@ def test_split_half(self):
raise unittest.SkipTest("This test is for a single worker only.")
+class FnApiBasedLullLoggingTest(unittest.TestCase):
+ def create_pipeline(self):
+return beam.Pipeline(
+runner=fn_api_runner.FnApiRunner(
+default_environment=beam_runner_api_pb2.Environment(
+urn=python_urns.EMBEDDED_PYTHON_GRPC),
+progress_request_frequency=0.5))
+
+ def test_lull_logging(self):
+
+if sys.version_info < (3, 4):
Review comment:
This is also for Python 2, that's right.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325416)
Time Spent: 1h 10m (was: 1h)
> Python SDK Worker should log lulls with progress-reporting thread
> -
>
> Key: BEAM-8333
> URL: https://issues.apache.org/jira/browse/BEAM-8333
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-harness
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[
https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325418=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325418
]
ASF GitHub Bot logged work on BEAM-8333:
Author: ASF GitHub Bot
Created on: 09/Oct/19 01:21
Start Date: 09/Oct/19 01:21
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #9746: [BEAM-8333]
Adding lull logging for SDK harness
URL: https://github.com/apache/beam/pull/9746#discussion_r332796254
##
File path: sdks/python/apache_beam/runners/worker/sdk_worker.py
##
@@ -403,10 +412,35 @@ def process_bundle_split(self, request, instruction_id):
instruction_id=instruction_id,
error='Instruction not running: %s' % instruction_id)
+ def _log_lull_in_bundle_processor(self, processor):
+state_sampler = processor.state_sampler
+sampler_info = state_sampler.get_info()
+if (sampler_info
+and sampler_info.time_since_transition
+and sampler_info.time_since_transition > self.log_lull_timeout_ns):
+ step_name = sampler_info.state_name.step_name
+ state_name = sampler_info.state_name.name
+ state_lull_log = (
+ 'There has been a processing lull of over %.2f seconds in state %s'
+ % (sampler_info.time_since_transition / 1e9, state_name))
+ step_name_log = (' in step %s ' % step_name) if step_name else ''
+
+ exec_thread = getattr(sampler_info, 'tracked_thread', None)
+ if exec_thread is not None:
+thread_frame = sys._current_frames().get(exec_thread.ident) # pylint:
disable=protected-access
+stack_trace = '\n'.join(
+traceback.format_stack(thread_frame)) if thread_frame else ''
+ else:
+stack_trace = '-NOT AVAILABLE-'
Review comment:
This should not occur, as the state sampler always marks the thread that
it's looking at - but I'd like to be cautious.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325418)
Time Spent: 1.5h (was: 1h 20m)
> Python SDK Worker should log lulls with progress-reporting thread
> -
>
> Key: BEAM-8333
> URL: https://issues.apache.org/jira/browse/BEAM-8333
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-harness
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[
https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325406=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325406
]
ASF GitHub Bot logged work on BEAM-8333:
Author: ASF GitHub Bot
Created on: 09/Oct/19 00:49
Start Date: 09/Oct/19 00:49
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #9746: [BEAM-8333]
Adding lull logging for SDK harness
URL: https://github.com/apache/beam/pull/9746#discussion_r332790525
##
File path: sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
##
@@ -1580,6 +1584,38 @@ def test_split_half(self):
raise unittest.SkipTest("This test is for a single worker only.")
+class FnApiBasedLullLoggingTest(unittest.TestCase):
+ def create_pipeline(self):
+return beam.Pipeline(
+runner=fn_api_runner.FnApiRunner(
+default_environment=beam_runner_api_pb2.Environment(
+urn=python_urns.EMBEDDED_PYTHON_GRPC),
+progress_request_frequency=0.5))
+
+ def test_lull_logging(self):
+
+if sys.version_info < (3, 4):
+ self.skipTest('Log-based assertions are supported after Python 3.4')
+try:
+ utils.check_compiled('apache_beam.runners.worker.opcounters')
+except RuntimeError:
+ self.skipTest('Cython is not available')
+
+with self.assertLogs(level='WARNING') as logs:
+ with self.create_pipeline() as p:
+sdk_worker.DEFAULT_LOG_LULL_TIMEOUT_NS = 1000 * 1000 # Lull after 1 ms
+
+_ = (p
+ | beam.Create([1])
+ | beam.Map(time.sleep))
+
+self.assertTrue(
Review comment:
Could we use assertRegexpMatches instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325406)
Time Spent: 50m (was: 40m)
> Python SDK Worker should log lulls with progress-reporting thread
> -
>
> Key: BEAM-8333
> URL: https://issues.apache.org/jira/browse/BEAM-8333
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-harness
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[
https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325407=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325407
]
ASF GitHub Bot logged work on BEAM-8333:
Author: ASF GitHub Bot
Created on: 09/Oct/19 00:49
Start Date: 09/Oct/19 00:49
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #9746: [BEAM-8333]
Adding lull logging for SDK harness
URL: https://github.com/apache/beam/pull/9746#discussion_r332791064
##
File path: sdks/python/apache_beam/runners/worker/sdk_worker.py
##
@@ -403,10 +412,35 @@ def process_bundle_split(self, request, instruction_id):
instruction_id=instruction_id,
error='Instruction not running: %s' % instruction_id)
+ def _log_lull_in_bundle_processor(self, processor):
+state_sampler = processor.state_sampler
+sampler_info = state_sampler.get_info()
+if (sampler_info
+and sampler_info.time_since_transition
+and sampler_info.time_since_transition > self.log_lull_timeout_ns):
+ step_name = sampler_info.state_name.step_name
+ state_name = sampler_info.state_name.name
+ state_lull_log = (
+ 'There has been a processing lull of over %.2f seconds in state %s'
+ % (sampler_info.time_since_transition / 1e9, state_name))
+ step_name_log = (' in step %s ' % step_name) if step_name else ''
+
+ exec_thread = getattr(sampler_info, 'tracked_thread', None)
+ if exec_thread is not None:
+thread_frame = sys._current_frames().get(exec_thread.ident) # pylint:
disable=protected-access
+stack_trace = '\n'.join(
+traceback.format_stack(thread_frame)) if thread_frame else ''
+ else:
+stack_trace = '-NOT AVAILABLE-'
Review comment:
When would this be the case?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325407)
Time Spent: 50m (was: 40m)
> Python SDK Worker should log lulls with progress-reporting thread
> -
>
> Key: BEAM-8333
> URL: https://issues.apache.org/jira/browse/BEAM-8333
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-harness
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[
https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325408=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325408
]
ASF GitHub Bot logged work on BEAM-8333:
Author: ASF GitHub Bot
Created on: 09/Oct/19 00:49
Start Date: 09/Oct/19 00:49
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #9746: [BEAM-8333]
Adding lull logging for SDK harness
URL: https://github.com/apache/beam/pull/9746#discussion_r332790861
##
File path: sdks/python/apache_beam/runners/worker/sdk_worker.py
##
@@ -403,10 +412,35 @@ def process_bundle_split(self, request, instruction_id):
instruction_id=instruction_id,
error='Instruction not running: %s' % instruction_id)
+ def _log_lull_in_bundle_processor(self, processor):
+state_sampler = processor.state_sampler
+sampler_info = state_sampler.get_info()
+if (sampler_info
+and sampler_info.time_since_transition
+and sampler_info.time_since_transition > self.log_lull_timeout_ns):
+ step_name = sampler_info.state_name.step_name
+ state_name = sampler_info.state_name.name
+ state_lull_log = (
+ 'There has been a processing lull of over %.2f seconds in state %s'
+ % (sampler_info.time_since_transition / 1e9, state_name))
+ step_name_log = (' in step %s ' % step_name) if step_name else ''
+
+ exec_thread = getattr(sampler_info, 'tracked_thread', None)
+ if exec_thread is not None:
+thread_frame = sys._current_frames().get(exec_thread.ident) # pylint:
disable=protected-access
+stack_trace = '\n'.join(
+traceback.format_stack(thread_frame)) if thread_frame else ''
+ else:
+stack_trace = '-NOT AVAILABLE-'
+
+ logging.warning(
+ '%s%s. Traceback:\n%s', state_lull_log, step_name_log, stack_trace)
+
def process_bundle_progress(self, request, instruction_id):
# It is an error to get progress for a not-in-flight bundle.
-processor = self.bundle_processor_cache.lookup(
-request.instruction_id)
+processor = self.bundle_processor_cache.lookup(request.instruction_id)
+if processor:
+ self._log_lull_in_bundle_processor(processor)
Review comment:
Do we know the performance impact? Especially since this will happen many
worker threads.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325408)
Time Spent: 1h (was: 50m)
> Python SDK Worker should log lulls with progress-reporting thread
> -
>
> Key: BEAM-8333
> URL: https://issues.apache.org/jira/browse/BEAM-8333
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-harness
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[
https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325409=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325409
]
ASF GitHub Bot logged work on BEAM-8333:
Author: ASF GitHub Bot
Created on: 09/Oct/19 00:49
Start Date: 09/Oct/19 00:49
Worklog Time Spent: 10m
Work Description: aaltay commented on pull request #9746: [BEAM-8333]
Adding lull logging for SDK harness
URL: https://github.com/apache/beam/pull/9746#discussion_r332790181
##
File path: sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
##
@@ -1580,6 +1584,38 @@ def test_split_half(self):
raise unittest.SkipTest("This test is for a single worker only.")
+class FnApiBasedLullLoggingTest(unittest.TestCase):
+ def create_pipeline(self):
+return beam.Pipeline(
+runner=fn_api_runner.FnApiRunner(
+default_environment=beam_runner_api_pb2.Environment(
+urn=python_urns.EMBEDDED_PYTHON_GRPC),
+progress_request_frequency=0.5))
+
+ def test_lull_logging(self):
+
+if sys.version_info < (3, 4):
Review comment:
SDK is not installable with < Python 3.4 anyway. You can probably drop this
(https://github.com/apache/beam/blob/master/sdks/python/setup.py#L182)
Or is this for protecting against py 2 users?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325409)
Time Spent: 1h (was: 50m)
> Python SDK Worker should log lulls with progress-reporting thread
> -
>
> Key: BEAM-8333
> URL: https://issues.apache.org/jira/browse/BEAM-8333
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-harness
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[ https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325370=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325370 ] ASF GitHub Bot logged work on BEAM-8333: Author: ASF GitHub Bot Created on: 08/Oct/19 22:36 Start Date: 08/Oct/19 22:36 Worklog Time Spent: 10m Work Description: pabloem commented on issue #9746: [BEAM-8333] Adding lull logging for SDK harness URL: https://github.com/apache/beam/pull/9746#issuecomment-539732787 Run Python 3.6 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 325370) Time Spent: 40m (was: 0.5h) > Python SDK Worker should log lulls with progress-reporting thread > - > > Key: BEAM-8333 > URL: https://issues.apache.org/jira/browse/BEAM-8333 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[ https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325340=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325340 ] ASF GitHub Bot logged work on BEAM-8333: Author: ASF GitHub Bot Created on: 08/Oct/19 21:36 Start Date: 08/Oct/19 21:36 Worklog Time Spent: 10m Work Description: pabloem commented on issue #9746: [BEAM-8333] Adding lull logging for SDK harness URL: https://github.com/apache/beam/pull/9746#issuecomment-539714669 Run Python 3.6 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 325340) Time Spent: 0.5h (was: 20m) > Python SDK Worker should log lulls with progress-reporting thread > - > > Key: BEAM-8333 > URL: https://issues.apache.org/jira/browse/BEAM-8333 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[ https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325327=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325327 ] ASF GitHub Bot logged work on BEAM-8333: Author: ASF GitHub Bot Created on: 08/Oct/19 20:54 Start Date: 08/Oct/19 20:54 Worklog Time Spent: 10m Work Description: pabloem commented on issue #9746: [BEAM-8333] Adding lull logging for SDK harness URL: https://github.com/apache/beam/pull/9746#issuecomment-539700024 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 325327) Time Spent: 20m (was: 10m) > Python SDK Worker should log lulls with progress-reporting thread > - > > Key: BEAM-8333 > URL: https://issues.apache.org/jira/browse/BEAM-8333 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8333) Python SDK Worker should log lulls with progress-reporting thread
[ https://issues.apache.org/jira/browse/BEAM-8333?focusedWorklogId=325316=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325316 ] ASF GitHub Bot logged work on BEAM-8333: Author: ASF GitHub Bot Created on: 08/Oct/19 20:30 Start Date: 08/Oct/19 20:30 Worklog Time Spent: 10m Work Description: pabloem commented on issue #9746: [BEAM-8333] Adding lull logging for SDK harness URL: https://github.com/apache/beam/pull/9746#issuecomment-539690884 This logs something like this: ``` WARNING:root:There has been a processing lull of over 0.45 seconds in state process-msecs in step Map(sleep) . Traceback: File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.6/concurrent/futures/thread.py", line 69, in _worker work_item.run() File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/google/home/pabloem/codes/log-lull-sdk/sdks/python/apache_beam/runners/worker/sdk_worker.py", line 203, in task self._execute(lambda: worker.do_instruction(work), work) File "/usr/local/google/home/pabloem/codes/log-lull-sdk/sdks/python/apache_beam/runners/worker/sdk_worker.py", line 170, in _execute response = task() File "/usr/local/google/home/pabloem/codes/log-lull-sdk/sdks/python/apache_beam/runners/worker/sdk_worker.py", line 203, in self._execute(lambda: worker.do_instruction(work), work) File "/usr/local/google/home/pabloem/codes/log-lull-sdk/sdks/python/apache_beam/runners/worker/sdk_worker.py", line 360, in do_instruction request.instruction_id) File "/usr/local/google/home/pabloem/codes/log-lull-sdk/sdks/python/apache_beam/runners/worker/sdk_worker.py", line 386, in process_bundle bundle_processor.process_bundle(instruction_id)) File "/usr/local/google/home/pabloem/codes/log-lull-sdk/sdks/python/apache_beam/runners/worker/bundle_processor.py", line 663, in process_bundle data.transform_id].process_encoded(data.data) File "/usr/local/google/home/pabloem/codes/log-lull-sdk/sdks/python/apache_beam/runners/worker/bundle_processor.py", line 143, in process_encoded self.output(decoded_value) File "/usr/local/google/home/pabloem/codes/log-lull-sdk/sdks/python/apache_beam/transforms/core.py", line 1356, in wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)] . ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 325316) Remaining Estimate: 0h Time Spent: 10m > Python SDK Worker should log lulls with progress-reporting thread > - > > Key: BEAM-8333 > URL: https://issues.apache.org/jira/browse/BEAM-8333 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
