reuvenlax commented on a change in pull request #15731:
URL: https://github.com/apache/beam/pull/15731#discussion_r737653840
##########
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
##########
@@ -716,13 +716,13 @@ private void addOutput(String name, PValue value,
Coder<?> valueCoder) {
translator.registerOutputName(value, name);
// If the output requires runner determined sharding, also append
necessary input properties.
- if (value instanceof PCollection
- &&
translator.runner.doesPCollectionRequireAutoSharding((PCollection<?>) value)) {
- addInput(PropertyNames.ALLOWS_SHARDABLE_STATE, "true");
- // Currently we only allow auto-sharding to be enabled through the
GroupIntoBatches
- // transform. So we also add the following property which
GroupIntoBatchesDoFn has, to allow
- // the backend to perform graph optimization.
- addInput(PropertyNames.PRESERVES_KEYS, "true");
+ if (value instanceof PCollection) {
+ if
(translator.runner.doesPCollectionRequireAutoSharding((PCollection<?>) value)) {
+ addInput(PropertyNames.ALLOWS_SHARDABLE_STATE, "true");
+ }
+ if (translator.runner.doesPCollectionPreserveKeys((PCollection<?>)
value)) {
+ addInput(PropertyNames.PRESERVES_KEYS, "true");
Review comment:
Yes - would need to separately update for portable.
##########
File path:
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
##########
@@ -1133,6 +1157,55 @@ private JobSpecification
runStreamingGroupIntoBatchesAndGetJobSpec(
pipeline, pipelineProto, sdkComponents, runner,
Collections.emptyList());
}
+ @Test
+ public void testBatchGroupIntoBatchesTranslation() throws Exception {
+ JobSpecification jobSpec =
+ runBatchGroupIntoBatchesAndGetJobSpec(false, Collections.emptyList());
+ List<Step> steps = jobSpec.getJob().getSteps();
+ Step shardedStateStep = steps.get(steps.size() - 1);
+ Map<String, Object> properties = shardedStateStep.getProperties();
+ assertTrue(properties.containsKey(PropertyNames.PRESERVES_KEYS));
+ assertEquals("true", getString(properties, PropertyNames.PRESERVES_KEYS));
+ }
+
+ @Test
+ public void testBatchGroupIntoBatchesWithShardedKeyTranslation() throws
Exception {
+ List<String> experiments = Collections.emptyList();
+ JobSpecification jobSpec = runBatchGroupIntoBatchesAndGetJobSpec(true,
experiments);
+ List<Step> steps = jobSpec.getJob().getSteps();
+ Step shardedStateStep = steps.get(steps.size() - 1);
+ Map<String, Object> properties = shardedStateStep.getProperties();
+ assertTrue(properties.containsKey(PropertyNames.PRESERVES_KEYS));
+ assertEquals("true", getString(properties, PropertyNames.PRESERVES_KEYS));
+ }
+
+ @Test
+ public void testBatchGroupIntoBatchesTranslationUnifiedWorker() throws
Exception {
Review comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
