[
https://issues.apache.org/jira/browse/BEAM-13067?focusedWorklogId=672919&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672919
]
ASF GitHub Bot logged work on BEAM-13067:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 02/Nov/21 00:20
Start Date: 02/Nov/21 00:20
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #15731:
URL: https://github.com/apache/beam/pull/15731#discussion_r740639484
##########
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
##########
@@ -716,13 +716,13 @@ private void addOutput(String name, PValue value,
Coder<?> valueCoder) {
translator.registerOutputName(value, name);
// If the output requires runner determined sharding, also append
necessary input properties.
- if (value instanceof PCollection
- &&
translator.runner.doesPCollectionRequireAutoSharding((PCollection<?>) value)) {
- addInput(PropertyNames.ALLOWS_SHARDABLE_STATE, "true");
- // Currently we only allow auto-sharding to be enabled through the
GroupIntoBatches
- // transform. So we also add the following property which
GroupIntoBatchesDoFn has, to allow
- // the backend to perform graph optimization.
- addInput(PropertyNames.PRESERVES_KEYS, "true");
+ if (value instanceof PCollection) {
+ if
(translator.runner.doesPCollectionRequireAutoSharding((PCollection<?>) value)) {
+ addInput(PropertyNames.ALLOWS_SHARDABLE_STATE, "true");
+ }
+ if (translator.runner.doesPCollectionPreserveKeys((PCollection<?>)
value)) {
+ addInput(PropertyNames.PRESERVES_KEYS, "true");
Review comment:
Would like to generalize and also make this a model feature. There are
some other things worth adding as well, such as PARTITIONED_BY_KEY (right now
the model doesn't really know if a PCollection<KV> is already partitioned by
key. runners can infer it by looking for a GroupByKey followed by only
key-preserving operations. however what is a source produces items already
partitioned by key?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 672919)
Time Spent: 1h 50m (was: 1h 40m)
> GroupIntoBatches should be considered a key-preserving operation by Dataflow
> ----------------------------------------------------------------------------
>
> Key: BEAM-13067
> URL: https://issues.apache.org/jira/browse/BEAM-13067
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Affects Versions: 2.33.0
> Reporter: Reuven Lax
> Priority: P2
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Dataflow today only marks GroupIntoBatches as key preserving if auto sharding
> is enabled. It should mark it as key preserving regardless.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
