[
https://issues.apache.org/jira/browse/BEAM-7421?focusedWorklogId=248881&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248881
]
ASF GitHub Bot logged work on BEAM-7421:
----------------------------------------
Author: ASF GitHub Bot
Created on: 27/May/19 14:39
Start Date: 27/May/19 14:39
Worklog Time Spent: 10m
Work Description: mxm commented on issue #8693: [BEAM-7421] Add missing
transform payload translator for Reshuffle
URL: https://github.com/apache/beam/pull/8693#issuecomment-496233982
@robertwb The composite transform (Reshuffle) is not changed, only its
resulting Flink/Spark translation is now as specified in the Runners'
translator classes. For non-portable pipelines, the pure existence of the
translator will cause Reshuffle to be recognized as a translateable transform.
For portable pipelines, it will actually set the `FunctionSpec` field of the
transform, while preserving the subtransforms.
Granted, this is a bit odd. Note that this only affects the Java SDK.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 248881)
Time Spent: 1h 10m (was: 1h)
> Flink/Spark non-portable runners doesn't override Reshuffle
> -----------------------------------------------------------
>
> Key: BEAM-7421
> URL: https://issues.apache.org/jira/browse/BEAM-7421
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark
> Reporter: Mike Kaplinskiy
> Assignee: Maximilian Michels
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Both the flink and the spark runners attempt to override the {{Reshuffle}}
> transform via the urn - e.g.
> [https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchTransformTranslators.java#L114]
> .
> Unfortunately there's no {{TransformPayloadTranslatorRegistrar}} for
> {{Reshuffle}}. This means the call to {{urnForTransformOrNull}} will always
> return {{null}} for a {{Reshuffle}} so they are never translated.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
