[
https://issues.apache.org/jira/browse/BEAM-11934?focusedWorklogId=565478&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-565478
]
ASF GitHub Bot logged work on BEAM-11934:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Mar/21 19:38
Start Date: 12/Mar/21 19:38
Worklog Time Spent: 10m
Work Description: chamikaramj commented on a change in pull request
#14164:
URL: https://github.com/apache/beam/pull/14164#discussion_r593405175
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java
##########
@@ -301,6 +321,21 @@
return toBuilder().setMaxNumWritersPerBundle(-1).build();
}
+ /**
+ * Returns a new {@link WriteFiles} that will write to the current {@link
FileBasedSink} with
+ * runner-determined sharding for unbounded data specifically. Currently
manual sharding is
+ * required for writing unbounded data with a fixed number of shards or a
predefined sharding
+ * function. This option allows the runners to get around that requirement
and perform automatic
+ * sharding.
+ *
+ * <p>Intended to only be used by runners. Users should use {@link
Review comment:
Ok, makes sense. I think a PTransformOverride should work for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 565478)
Time Spent: 1h 40m (was: 1.5h)
> GCS streaming file sink uses runner determined sharding
> -------------------------------------------------------
>
> Key: BEAM-11934
> URL: https://issues.apache.org/jira/browse/BEAM-11934
> Project: Beam
> Issue Type: Improvement
> Components: io-java-files, io-java-gcp
> Reporter: Siyuan Chen
> Assignee: Siyuan Chen
> Priority: P1
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Integrate file sink with shardable `GroupIntoBatches` (BEAM-10475) to allow
> runner determined dynamic sharding for streaming use cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
