Polber commented on PR #30864:
URL: https://github.com/apache/beam/pull/30864#issuecomment-2040341056
> @robertwb After further investigation, it appears this occurs anytime that
2 subsequent ExternalTransforms are declared that use the same gradle target,
and the first is given multiple inputs (PCollectionTuple). I haven't been able
to repro unless the input to the first ExternalTransform is a dict of
PCollections.
Note: changing the gradle target for `LogForTesting` on either example I
posted will result in a successful run.
The transforms also do not have to be connected to each other:
```
service = external.JavaJarExpansionService(
subprocess_server.JavaJarServer.path_to_beam_jar(
gradle_target='sdks:java:extensions:sql:expansion-service:shadowJar',
artifact_id=None
)
)
result = {'i1': i1, 'i2': i2} | 'Sql1' >> external.ExternalTransform(
'beam:external:java:sql:v1',
external.ImplicitSchemaPayloadBuilder(
{'query': 'SELECT * FROM i1 INNER JOIN i2 ON i1.id = i2.id'}
).payload(),
service)
result2 = {'i1': i1, 'i2': i2} | 'Flatten' >>
external.SchemaAwareExternalTransform(
'beam:schematransform:org.apache.beam:yaml:flatten:v1',
service, rearrange_based_on_discovery=True)
```
and it only fails if one transform is `ExternalTransform` and one is
`SchemaAwareExternalTransform`. i.e. the following does not fail:
```
service = external.JavaJarExpansionService(
subprocess_server.JavaJarServer.path_to_beam_jar(
gradle_target='sdks:java:extensions:sql:expansion-service:shadowJar',
artifact_id=None
)
)
result2 = {'i1': i1, 'i2': i2} | 'Flatten' >>
external.SchemaAwareExternalTransform(
'beam:schematransform:org.apache.beam:yaml:flatten:v1',
service, rearrange_based_on_discovery=True
) | 'LogForTesting' >> external.SchemaAwareExternalTransform(
'beam:schematransform:org.apache.beam:yaml:log_for_testing:v1',
service, rearrange_based_on_discovery=True)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
