kennknowles commented on issue #19239: URL: https://github.com/apache/beam/issues/19239#issuecomment-1156885794
The PTransforms in a graph contain "user code" in the form of Java and Python DoFns (things like BoundedSource and UnboundedSource also are DoFns when you get to execution). A connector like BigQueryIO is "just" a PTransform, in other words a small subgraph that performance the data reading. For most connectors, we expect to have just one implementation. Usually this will probably be Java. Other SDKs will use the connector via Beam's portability APIs. But some connectors are implemented more than once, in different SDKs. For example FileIO is implemented in all SDKs largely to get the SDK started. Another example is BigQueryIO which is implemented in both Java and Python. So the DoFns that are executed in each case are different. The Python implementation of BigQueryIO is also missing features that are available in Java. Hope that helps! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
