lostluck opened a new issue, #31991: URL: https://github.com/apache/beam/issues/31991
Java PipelineTests are failing in some cases with PTransformSpec.Urns set with "ProjectTag" and "IdentityTransform". It's not clear what's going on here, except that these are unknown/non-standard URNs being used outside of a composite. Iterate by setting as the pipeline test, then running the Gradle command, pointed at a prism instance. TEST=org.apache.beam.sdk.PipelineTest ./gradlew :runners:portability:java:ulrLoopbackValidatesRunnerTests -PjobEndpoint=localhost:8073 --tests="$TEST" There should be at least 3 failing tests. testIdentityTransform, testTupleProjectionTransform, and testTupleInjectionTransform. It's possible the "correct" move is to allow arbitrary URNs, and treat them as the DoFn urn, but this feels sub optimal for respecting new SDKs. Examining the "Identity" transform, it looks like it's a Composite that has no sub transforms, which means we're not doing the "right" thing WRT detecting a composite, assuming they always have sub transforms. Similarly, "ProjectTag", has no transforms, but only extracts out (or even wraps) a pcollection out from a Pcollection tuple (or creates one). ----- The resolution is to allow this style of "composites", and assume unknown URNs are of this style of Composite transforms. This means no longer requiring composites to have sub transforms, and ensure that the no-op (input is output), and tag filter (output is subset of output) are allowed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
