Luke, I studied the resources you provided. However, it's still a little unclear to me as to how I could use the BigTableIO <https://beam.apache.org/releases/javadoc/2.1.0/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.html> in Java from a Python pipeline. The examples and documentation first implement a demo class in Java and then show how to use it.
I was wondering if there was a guide on using the existing connectors (i.e., without defining them first) from Python pipelines. I am probably mistaken somewhere so happy to rectify myself if that's the case. Sayak Paul | sayak.dev On Thu, Jan 6, 2022 at 10:35 PM Sayak Paul <[email protected]> wrote: > Thanks! > > On Thu, 6 Jan, 2022, 22:27 Luke Cwik, <[email protected]> wrote: > >> +1 on using cross language to get the Java Bigtable connector that >> already exists. >> >> You could also take a look at this other xlang documentation[1] and look >> at an existing implementation such as kafka[2] that is xlang. >> >> Finally there was support added to use many transforms in Java using the >> class name and builder methods[3]. >> >> 1: https://beam.apache.org/documentation/patterns/cross-language/ >> 2: >> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/kafka.py >> 3: https://issues.apache.org/jira/browse/BEAM-12769 >> >> >> On Thu, Jan 6, 2022 at 4:41 AM Sayak Paul <[email protected]> wrote: >> >>> Hi folks, >>> >>> My project needs reading data from Cloud BigTable. We are aware that an >>> IO connector for BigTable is available in the Java SDK. So we could >>> probably make use of the cross-language capabilities >>> <https://beam.apache.org/documentation/programming-guide/#1311-creating-cross-language-java-transforms> >>> of Beam and make it work. I am, however, looking for >>> guidance/resources/pointers that could be beneficial to build a Beam >>> pipeline in Python that reads data from Cloud BigTable. Any relevant clue >>> would be greatly appreciated. >>> >>> Sayak Paul | sayak.dev >>> >>>
