On Mon, Jul 25, 2022 at 12:53 PM Lina Mårtensson via dev < dev@beam.apache.org> wrote:
> Hi dev, > > We're starting to incorporate BigTable in our stack and I've delighted > my co-workers with how easy it was to create some BigTables with > Beam... but there doesn't appear to be a reader for BigTable in > Python. > > First off, is there a good reason why not/any reason why it would be > difficult? > There's was a previous effort to implement a Python BT source but that was not completed: https://github.com/apache/beam/pull/11295#issuecomment-646378304 > > I could write one, but before I start, I'd love some input to make it > easier. > > It appears that there would be two options: either write one in > Python, or try to set one up with x-language from Java which I see is > done e.g. with the Spanner IO Connector. > Any recommendation on which one to pick or potential pitfalls in either > choice? > > If I write one in Python, what should I think about? > It is not obvious to me how to achieve parallelization, so any tips > here would be welcome. > I would strongly prefer developing a Python wrapper for the existing Java BT source using Beam's Multi-language Pipelines framework over developing a new Python source. https://beam.apache.org/documentation/programming-guide/#multi-language-pipelines Thanks, Cham > > Thanks! > -Lina >
