@lukecwik this is probably something we should take to the mailing list for broader visibility. Currently the Flink runner does the pipeline surgery before translation. With the proposed change the customization referenced above would need to spread over multiple components (fuser/prep and translator) while it's encapsulated in a single place now.
In the example the Flink sources really are primitives that should remain handled on the runner side as such (effectively like ParDo without inputs). Would it perhaps also make sense to assume that as a default when our common fusion code does not recognize a URN? If I was to introduce a custom URN that requires fusion break, then yes, it would be necessary to provide that information to the fuser. That seems cleaner than the runner temporarily switching URNs to appease the fuser. [ Full content available at: https://github.com/apache/beam/pull/6328 ] This message was relayed via gitbox.apache.org for [email protected]
