[
https://issues.apache.org/jira/browse/BEAM-5026?focusedWorklogId=162327&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-162327
]
ASF GitHub Bot logged work on BEAM-5026:
----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Nov/18 14:53
Start Date: 04/Nov/18 14:53
Worklog Time Spent: 10m
Work Description: tweise opened a new pull request #6936: [BEAM-5026]
Workaround for portable batch wordcount example file write in Docker container.
URL: https://github.com/apache/beam/pull/6936
Workaround for portable batch wordcount example file write in Docker
container.
------------------------
Follow this checklist to help us incorporate your contribution quickly and
easily:
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone
(e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
| --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 162327)
Time Spent: 10m
Remaining Estimate: 0h
> Portable flink wordcount fails sometimes due to non-existent source path in
> FileBasedSink._check_state_for_finalize_write
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-5026
> URL: https://issues.apache.org/jira/browse/BEAM-5026
> Project: Beam
> Issue Type: Bug
> Components: examples-python, runner-flink, sdk-py-core
> Affects Versions: 2.6.0
> Reporter: Ryan Williams
> Assignee: Ankur Goenka
> Priority: Minor
> Fix For: 2.7.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Running portable flink wordcount locally:
> In one terminal:
> {code:java}
> ./gradlew :beam-runners-flink_2.11-job-server:runShadow{code}
> In another:
> {code:java}
> python -m apache_beam.examples.wordcount --harness_docker_image <image>
> --input /etc/profile --output /tmp/py-wordcount-direct
> --experiments=beam_fn_api --runner=PortableRunner
> --job_endpoint=localhost:8099 --sdk_location=container{code}
> Typically, the first time I run this for a given job-server instance, I see a
> failure like this ([full
> output|https://gist.github.com/ryan-williams/a96bf259898b6260cd4f00b8a232057c#file-gistfile1-txt-L3460]):
> {code:java}
> File "apache_beam/runners/common.py", line 661, in
> apache_beam.runners.common._OutputProcessor.process_outputs
> def process_outputs(self, windowed_input_element, results):
> File "apache_beam/runners/common.py", line 676, in
> apache_beam.runners.common._OutputProcessor.process_outputs
> for result in results:
> File "/usr/local/lib/python2.7/site-packages/apache_beam/io/iobase.py", line
> 1074, in <genexpr>
> return (window.TimestampedValue(v, window.MAX_TIMESTAMP) for v in outputs)
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/io/filebasedsink.py",
> line 271, in finalize_write
> self._check_state_for_finalize_write(writer_results, num_shards))
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/io/filebasedsink.py",
> line 249, in _check_state_for_finalize_write
> src, dst))
> BeamIOError: src and dst files do not exist. src:
> /tmp/beam-temp-py-wordcount-direct-6a0d8862908c11e88de8025000000001/5cfa9f22-9246-41fb-adef-ca04d5a5fe50.py-wordcount-direct,
> dst: /tmp/py-wordcount-direct-00000-of-00001 with exceptions None [while
> running 'write/Write/WriteImpl/FinalizeWrite'] with exceptions None
> {code}
> This is after a fix to [a slightly earlier failure in {{FileBasedSink}}
> documented on
> BEAM-4742|https://issues.apache.org/jira/browse/BEAM-4742?focusedCommentId=16545622&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16545622]
> which I've been working on in
> [#5903|https://github.com/apache/beam/pull/5903].
> It typically occurs only on the first run of wordcount against a given
> job-server instance.
> I'm curious whether others see this, whether it's some race condition in the
> FileBasedSink, LocalFileSystem, my macbook's disk, or somewhere else, or
> whether some temporary directory is getting created on the first run (for
> each job-server) that explains why subsequent wordcount runs succeed, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
