[
https://issues.apache.org/jira/browse/BEAM-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16788434#comment-16788434
]
Chamikara Jayalath commented on BEAM-6683:
------------------------------------------
I agree that Schemas probably should be the data translation mechanism across
languages once we have that.
Till then probably we should be able to use a coder for a language neutral
format (for example, ProtoCoder) for testing.
Something like
pcollection =
pipeline.apply(StubforJavaIO()).apply(StubForJavaTransformThatProducesProtos()).apply(PythonTransformThatConsumesProtos()).apply(AnotherPythonTransform())
Would you mind replying to the dev list thread
(https://lists.apache.org/thread.html/b99ba8527422e31ec7bb7ad9dc3a6583551ea392ebdc5527b5fb4a67@%3Cdev.beam.apache.org%3E)
with possible solutions here ?
> Add an integration test suite for cross-language transforms for Flink runner
> ----------------------------------------------------------------------------
>
> Key: BEAM-6683
> URL: https://issues.apache.org/jira/browse/BEAM-6683
> Project: Beam
> Issue Type: Test
> Components: testing
> Reporter: Chamikara Jayalath
> Assignee: Heejong Lee
> Priority: Major
>
> We should add an integration test suite that covers following.
> (1) Currently available Java IO connectors that do not use UDFs work for
> Python SDK on Flink runner.
> (2) Currently available Python IO connectors that do not use UDFs work for
> Java SDK on Flink runner.
> (3) Currently available Java/Python pipelines work in a scalable manner for
> cross-language pipelines (for example, try 10GB, 100GB input for
> textio/avroio for Java and Python).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
