[
https://issues.apache.org/jira/browse/BEAM-6242?focusedWorklogId=177029&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177029
]
ASF GitHub Bot logged work on BEAM-6242:
----------------------------------------
Author: ASF GitHub Bot
Created on: 19/Dec/18 15:05
Start Date: 19/Dec/18 15:05
Worklog Time Spent: 10m
Work Description: mxm closed pull request #7298: [BEAM-6242] Fix Portable
WordCount PreCommit by caching environment
URL: https://github.com/apache/beam/pull/7298
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index db18b2b133be..59d79ab6ff38 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -222,11 +222,16 @@ def portableWordCountTask(name, streaming) {
"--runner=PortableRunner",
]
+ // Workaround for ensuring writes to the local file system
+ // stay within the same Docker container. Otherwise the Write
+ // logic would throw because files can't be shared between
+ // different executable stages.
+ options += ["--environment_cache_millis=60000"]
+
if (streaming) {
options += ["--streaming"]
+ options += ["--parallelism=8"]
} else {
- // workaround for local file output in docker container
- options += ["--environment_cache_millis=10000"]
// [BEAM-5167] Workaround for scheduling issue between SDKHarness and
Flink
options += ["--parallelism=1"]
}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 177029)
Time Spent: 3h 10m (was: 3h)
> PreCommit Portable WordCount is flaky
> -------------------------------------
>
> Key: BEAM-6242
> URL: https://issues.apache.org/jira/browse/BEAM-6242
> Project: Beam
> Issue Type: Test
> Components: runner-flink
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/42/consoleText]
>
> {noformat}
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: Error received from SDK harness for instruction
> 533: Traceback (most recent call last):
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 145, in _execute
> response = task()
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 180, in <lambda>
> self._execute(lambda: worker.do_instruction(work), work)
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 253, in do_instruction
> request.instruction_id)
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 269, in process_bundle
> bundle_processor.process_bundle(instruction_id)
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 489, in process_bundle
> ].process_encoded(data.data)
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 126, in process_encoded
> self.output(decoded_value)
> File "apache_beam/runners/worker/operations.py", line 182, in
> apache_beam.runners.worker.operations.Operation.output
> def output(self, windowed_value, output_index=0):
> File "apache_beam/runners/worker/operations.py", line 183, in
> apache_beam.runners.worker.operations.Operation.output
> cython.cast(Receiver,
> self.receivers[output_index]).receive(windowed_value)
> File "apache_beam/runners/worker/operations.py", line 89, in
> apache_beam.runners.worker.operations.ConsumerSet.receive
> cython.cast(Operation, consumer).process(windowed_value)
> File "apache_beam/runners/worker/operations.py", line 497, in
> apache_beam.runners.worker.operations.DoOperation.process
> with self.scoped_process_state:
> File "apache_beam/runners/worker/operations.py", line 498, in
> apache_beam.runners.worker.operations.DoOperation.process
> self.dofn_receiver.receive(o)
> File "apache_beam/runners/common.py", line 680, in
> apache_beam.runners.common.DoFnRunner.receive
> self.process(windowed_value)
> File "apache_beam/runners/common.py", line 686, in
> apache_beam.runners.common.DoFnRunner.process
> self._reraise_augmented(exn)
> File "apache_beam/runners/common.py", line 724, in
> apache_beam.runners.common.DoFnRunner._reraise_augmented
> raise_with_traceback(new_exn)
> File "apache_beam/runners/common.py", line 684, in
> apache_beam.runners.common.DoFnRunner.process
> self.do_fn_invoker.invoke_process(windowed_value)
> File "apache_beam/runners/common.py", line 535, in
> apache_beam.runners.common.PerWindowInvoker.invoke_process
> self._invoke_per_window(
> File "apache_beam/runners/common.py", line 604, in
> apache_beam.runners.common.PerWindowInvoker._invoke_per_window
> output_processor.process_outputs(
> File "apache_beam/runners/common.py", line 755, in
> apache_beam.runners.common._OutputProcessor.process_outputs
> def process_outputs(self, windowed_input_element, results):
> File "apache_beam/runners/common.py", line 770, in
> apache_beam.runners.common._OutputProcessor.process_outputs
> for result in results:
> File "/usr/local/lib/python2.7/site-packages/apache_beam/io/iobase.py",
> line 1077, in <genexpr>
> window.TimestampedValue(v, timestamp.MAX_TIMESTAMP) for v in outputs)
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/io/filebasedsink.py",
> line 321, in finalize_write
> 'Encountered exceptions in finalize_write: %s' % all_exceptions)
> Exception: Encountered exceptions in finalize_write: [IOError(OSError(2, 'No
> such file or directory'),), IOError(OSError(2, 'No such file or
> directory'),)] [while running 'write/Write/WriteImpl/FinalizeWrite']
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57)
> at
> org.apache.beam.runners.fnexecution.control.SdkHarnessClient$ActiveBundle.close(SdkHarnessClient.java:263)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator$SdkHarnessDoFnRunner.finishBundle(ExecutableStageDoFnOperator.java:540)
> ... 14 more{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
