On Thu, Aug 4, 2022 at 5:29 PM Daniel Collins <[email protected]> wrote:
> Hello Drew, > > > I upgraded to apache-beam 2.40.0 and tried to access > apache_beam.io.gcp.pubsublite.ReadFromPubSubLite > > You should ensure to import `apache_beam.io.gcp.pubsublite.*`. I have no > idea why the specific import isn't working- but that should work. If > its not, I'll look into it more. > > > writing native Spark code to pull from PubSub Lite > > Note that we have a spark native source you can use. I'm unsure if spark > works with beam python however, Chamikara would know that better. > https://github.com/googleapis/java-pubsublite-spark > It should be supported. See instructions here under "Portable (Java/Python/Go)": https://beam.apache.org/documentation/runners/spark/ > > > -Daniel > > On Thu, Aug 4, 2022 at 7:48 PM Drew Forbes < > [email protected]> wrote: > >> I've actually not used PyBeam, I just meant writing Beam code with >> Python. Didn't realize there was a whole separate PyBeam package. >> > Thanks for clarifying. Thanks, Cham > >> I feel dumb asking, but basically we just couldn't get the import to >> work. I upgraded to apache-beam 2.40.0 and tried to access apache_beam.io >> .gcp.pubsublite.ReadFromPubSubLite through various means (regular >> import, proto_api, something like .external., etc) within Python and >> determined that there just wasn't anything to access. We could definitely >> have been wrong about that but it wasn't clear how to move forward so we >> just switched our focus to writing native Spark code to pull from PubSub >> Lite >> >> On Thu, Aug 4, 2022 at 6:46 PM Chamikara Jayalath <[email protected]> >> wrote: >> >>> I believe this should be fully working. I'm not familiar with PyBeam >>> though. Is the execution mechanism the same as running a regular Beam >>> pipeline ? Also, note that for multi-language, you need to use a portable >>> Beam runner. >>> >>> +Daniel Collins <[email protected]> who implemented this. >>> >>> Thanks, >>> Cham >>> >>> On Thu, Aug 4, 2022 at 11:24 AM Austin Bennett < >>> [email protected]> wrote: >>> >>>> Hi Users/Devs, >>>> >>>> Drew, copied, reported having troubles with PubSub Lite: >>>> >>>> "we just weren’t able to get PubSub Lite working with PyBeam. It’s been >>>> a few weeks since we last tried, but we were just trying to use >>>> `apache_beam.io.gcp.pubsublite.ReadFromPubSubLite` (here >>>> <https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.pubsublite.html> >>>> ) in PyBeam and couldn’t get it to import so we just gave up. >From the >>>> looks of the repo we couldn’t tell if it was ever actually fully >>>> implemented and published" >>>> >>>> I haven't used myself, and figured others might be able to >>>> comment/share at least if any have had success using and/or at least >>>> whether fully tested/implemented IO ( whether available via cross-language >>>> or 'native' python ). >>>> >>>> Please share any thoughts here. >>>> >>>> Cheers, >>>> Austin >>>> >>>>
