[
https://issues.apache.org/jira/browse/BEAM-13067?focusedWorklogId=670863&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670863
]
ASF GitHub Bot logged work on BEAM-13067:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Oct/21 16:36
Start Date: 27/Oct/21 16:36
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #15731:
URL: https://github.com/apache/beam/pull/15731#discussion_r737653840
##########
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
##########
@@ -716,13 +716,13 @@ private void addOutput(String name, PValue value,
Coder<?> valueCoder) {
translator.registerOutputName(value, name);
// If the output requires runner determined sharding, also append
necessary input properties.
- if (value instanceof PCollection
- &&
translator.runner.doesPCollectionRequireAutoSharding((PCollection<?>) value)) {
- addInput(PropertyNames.ALLOWS_SHARDABLE_STATE, "true");
- // Currently we only allow auto-sharding to be enabled through the
GroupIntoBatches
- // transform. So we also add the following property which
GroupIntoBatchesDoFn has, to allow
- // the backend to perform graph optimization.
- addInput(PropertyNames.PRESERVES_KEYS, "true");
+ if (value instanceof PCollection) {
+ if
(translator.runner.doesPCollectionRequireAutoSharding((PCollection<?>) value)) {
+ addInput(PropertyNames.ALLOWS_SHARDABLE_STATE, "true");
+ }
+ if (translator.runner.doesPCollectionPreserveKeys((PCollection<?>)
value)) {
+ addInput(PropertyNames.PRESERVES_KEYS, "true");
Review comment:
Yes - would need to separately update for portable.
##########
File path:
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
##########
@@ -1133,6 +1157,55 @@ private JobSpecification
runStreamingGroupIntoBatchesAndGetJobSpec(
pipeline, pipelineProto, sdkComponents, runner,
Collections.emptyList());
}
+ @Test
+ public void testBatchGroupIntoBatchesTranslation() throws Exception {
+ JobSpecification jobSpec =
+ runBatchGroupIntoBatchesAndGetJobSpec(false, Collections.emptyList());
+ List<Step> steps = jobSpec.getJob().getSteps();
+ Step shardedStateStep = steps.get(steps.size() - 1);
+ Map<String, Object> properties = shardedStateStep.getProperties();
+ assertTrue(properties.containsKey(PropertyNames.PRESERVES_KEYS));
+ assertEquals("true", getString(properties, PropertyNames.PRESERVES_KEYS));
+ }
+
+ @Test
+ public void testBatchGroupIntoBatchesWithShardedKeyTranslation() throws
Exception {
+ List<String> experiments = Collections.emptyList();
+ JobSpecification jobSpec = runBatchGroupIntoBatchesAndGetJobSpec(true,
experiments);
+ List<Step> steps = jobSpec.getJob().getSteps();
+ Step shardedStateStep = steps.get(steps.size() - 1);
+ Map<String, Object> properties = shardedStateStep.getProperties();
+ assertTrue(properties.containsKey(PropertyNames.PRESERVES_KEYS));
+ assertEquals("true", getString(properties, PropertyNames.PRESERVES_KEYS));
+ }
+
+ @Test
+ public void testBatchGroupIntoBatchesTranslationUnifiedWorker() throws
Exception {
Review comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 670863)
Time Spent: 50m (was: 40m)
> GroupIntoBatches should be considered a key-preserving operation by Dataflow
> ----------------------------------------------------------------------------
>
> Key: BEAM-13067
> URL: https://issues.apache.org/jira/browse/BEAM-13067
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Affects Versions: 2.33.0
> Reporter: Reuven Lax
> Priority: P2
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Dataflow today only marks GroupIntoBatches as key preserving if auto sharding
> is enabled. It should mark it as key preserving regardless.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
