[
https://issues.apache.org/jira/browse/BEAM-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083336#comment-16083336
]
ASF GitHub Bot commented on BEAM-2601:
--------------------------------------
GitHub user reuvenlax opened a pull request:
https://github.com/apache/beam/pull/3546
[BEAM-2601] Fix broken per-destination finalization.
We now finalize each destination separately. Since temporary-file cleanup
happens (for the non-windowed case) by deleting the entire temp directory, we
delay cleanup until all destinations have been finalized.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/reuvenlax/incubator-beam
fix_dynamic_destination_sharding
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3546.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3546
----
commit 7254dcf312251a462c70b24c8ff8861c2f009b2b
Author: Reuven Lax <[email protected]>
Date: 2017-07-12T02:19:42Z
Fix per-destination finalization.
----
> FileBasedSink produces incorrect shards when writing to multiple destinations
> -----------------------------------------------------------------------------
>
> Key: BEAM-2601
> URL: https://issues.apache.org/jira/browse/BEAM-2601
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Reuven Lax
> Assignee: Davor Bonaci
> Fix For: 2.2.0
>
>
> FileBasedSink now supports multiple dynamic destinations, however it
> finalizes all files in a bundle without paying attention to destination. This
> means that the shard counts will be incorrect across these destinations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
