boyuanzz commented on a change in pull request #13805:
URL: https://github.com/apache/beam/pull/13805#discussion_r569671565
##########
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##########
@@ -1365,21 +1365,22 @@ void
addPCollectionRequiringIndexedFormat(PCollection<?> pcol) {
}
void maybeRecordPCollectionWithAutoSharding(PCollection<?> pcol) {
- if (hasExperiment(options, "beam_fn_api")) {
- LOG.warn(
- "Runner determined sharding not available in Dataflow for
GroupIntoBatches for portable "
- + "jobs. Default sharding will be applied.");
- return;
- }
- if (!options.isEnableStreamingEngine()) {
- LOG.warn(
- "Runner determined sharding not available in Dataflow for
GroupIntoBatches for Streaming "
- + "Appliance jobs. Default sharding will be applied.");
- return;
- }
- if (hasExperiment(options, "enable_streaming_auto_sharding")) {
- pcollectionsRequiringAutoSharding.add(pcol);
- }
+ // Auto-sharding is only supported in Streaming Engine.
+ checkArgument(
+ options.isEnableStreamingEngine(),
+ "Runner determined sharding not available in Dataflow for
GroupIntoBatches for"
+ + " non-Streaming-Engine jobs.");
Review comment:
I'm thinking maybe that we give the full instructions will be more
helpful. For example, `In order to use runenr determined sharding, please use
--experiments=enable_streaming_auto_sharding, --experiments=use_runner_v2,
--streaming, --enable_streaming_engine`. But it really depends on you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
