[
https://issues.apache.org/jira/browse/BEAM-5455?focusedWorklogId=146605&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146605
]
ASF GitHub Bot logged work on BEAM-5455:
----------------------------------------
Author: ASF GitHub Bot
Created on: 21/Sep/18 21:49
Start Date: 21/Sep/18 21:49
Worklog Time Spent: 10m
Work Description: tweise closed pull request #6459: [BEAM-5455] Log
per-bundle messages at debug in python sdk
URL: https://github.com/apache/beam/pull/6459
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/apache_beam/runners/worker/bundle_processor.py
b/sdks/python/apache_beam/runners/worker/bundle_processor.py
index ae636607c66..6b633576020 100644
--- a/sdks/python/apache_beam/runners/worker/bundle_processor.py
+++ b/sdks/python/apache_beam/runners/worker/bundle_processor.py
@@ -376,7 +376,7 @@ def process_bundle(self, instruction_id):
self.state_sampler.start()
# Start all operations.
for op in reversed(self.ops.values()):
- logging.info('start %s', op)
+ logging.debug('start %s', op)
op.start()
# Inject inputs from data plane.
@@ -388,7 +388,7 @@ def process_bundle(self, instruction_id):
# Finish all operations.
for op in self.ops.values():
- logging.info('finish %s', op)
+ logging.debug('finish %s', op)
op.finish()
finally:
self.state_sampler.stop_if_still_running()
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker.py
b/sdks/python/apache_beam/runners/worker/sdk_worker.py
index a97e4242f93..7c6054147a6 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker.py
@@ -109,7 +109,7 @@ def get_responses():
yield response
for work_request in control_stub.Control(get_responses()):
- logging.info('Got work %s', work_request.instruction_id)
+ logging.debug('Got work %s', work_request.instruction_id)
request_type = work_request.WhichOneof('request')
# Name spacing the request method with 'request_'. The called method
# will be like self.request_register(request)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 146605)
Time Spent: 1h (was: 50m)
> Don't info log for every bundle in the python sdk
> -------------------------------------------------
>
> Key: BEAM-5455
> URL: https://issues.apache.org/jira/browse/BEAM-5455
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-harness
> Reporter: Micah Wylde
> Assignee: Micah Wylde
> Priority: Minor
> Labels: portability-flink
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently the python sdk logs a bunch of messages at INFO level for every
> bundle. For the portable flink runner, which uses a bundle size of 1 in
> streaming mode, this produces huge log spew:
> {code:java}
> [grpc-default-executor-1] INFO sdk_worker.run - Got work 4
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start
> <DataOutputOperation >
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start
> <DoOperation window output_tags=['out']>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start
> <DoOperation pair_with_one output_tags=['out']>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start
> <DoOperation timestamp output_tags=['out']>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start
> <DoOperation decode output_tags=['out']>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start
> <DataInputOperation receivers=[ConsumerSet[.out0,
> coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish
> <DataInputOperation receivers=[ConsumerSet[.out0,
> coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish
> <DoOperation decode output_tags=['out'], receivers=[ConsumerSet[decode.out0,
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish
> <DoOperation timestamp output_tags=['out'],
> receivers=[ConsumerSet[timestamp.out0,
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish
> <DoOperation pair_with_one output_tags=['out'],
> receivers=[ConsumerSet[pair_with_one.out0,
> coder=WindowedValueCoder[TupleCoder[FastPrimitivesCoder,
> FastPrimitivesCoder]], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish
> <DoOperation window output_tags=['out'], receivers=[ConsumerSet[window.out0,
> coder=WindowedValueCoder[TupleCoder[FastPrimitivesCoder,
> FastPrimitivesCoder]], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish
> <DataOutputOperation >
> {code}
>
> These log messages should probably be at debug instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
