nehsyc commented on a change in pull request #13805:
URL: https://github.com/apache/beam/pull/13805#discussion_r568357423
##########
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##########
@@ -1365,21 +1365,27 @@ void
addPCollectionRequiringIndexedFormat(PCollection<?> pcol) {
}
void maybeRecordPCollectionWithAutoSharding(PCollection<?> pcol) {
- if (hasExperiment(options, "beam_fn_api")) {
+ if (!options.isEnableStreamingEngine()) {
LOG.warn(
- "Runner determined sharding not available in Dataflow for
GroupIntoBatches for portable "
- + "jobs. Default sharding will be applied.");
+ "Runner determined sharding not available in Dataflow for
GroupIntoBatches for "
+ + "non-Streaming Engine jobs. Default sharding will be
applied.");
return;
}
- if (!options.isEnableStreamingEngine()) {
+ if (!hasExperiment(options, "enable_streaming_auto_sharding")) {
LOG.warn(
- "Runner determined sharding not available in Dataflow for
GroupIntoBatches for Streaming "
- + "Appliance jobs. Default sharding will be applied.");
+ "Runner determined sharding not enabled in Dataflow for
GroupIntoBatches for Streaming "
+ + "Engine jobs: --enable_streaming_auto_sharding=false. Default
sharding will be "
+ + "applied.");
return;
}
- if (hasExperiment(options, "enable_streaming_auto_sharding")) {
- pcollectionsRequiringAutoSharding.add(pcol);
+ if (hasExperiment(options, "beam_fn_api") && !hasExperiment(options,
"use_runner_v2")) {
+ LOG.warn(
+ "Runner determined sharding not available in Dataflow for
GroupIntoBatches for portable "
+ + "jobs not using runner v2: --beam_fn_api=true,
--use_runner_v2=false. Default "
+ + "sharding will be applied.");
+ return;
Review comment:
Actually on a second thought, do we have to reject JRH? I guess it can
work (although I haven't verified). Would it be a better user experience if we
fail the job when autosharding falls back to the default implementation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
