@cham thanks for bringing the conversation back to the list ( esp. for anyone else searching/wondering in the future )!
>From what I understand/summary: Python should be able to call via X-Lang the [ Java ] PubSubLite IO for use with any underlying runner ( well, that utilizes portable runner, ex: Spark, Flink, DataflowV2, etc ) On Thu, Aug 4, 2022 at 5:49 PM Chamikara Jayalath via user < [email protected]> wrote: > > > On Thu, Aug 4, 2022 at 5:29 PM Daniel Collins <[email protected]> > wrote: > >> Hello Drew, >> >> > I upgraded to apache-beam 2.40.0 and tried to access >> apache_beam.io.gcp.pubsublite.ReadFromPubSubLite >> >> You should ensure to import `apache_beam.io.gcp.pubsublite.*`. I have no >> idea why the specific import isn't working- but that should work. If >> its not, I'll look into it more. >> >> > writing native Spark code to pull from PubSub Lite >> >> Note that we have a spark native source you can use. I'm unsure if spark >> works with beam python however, Chamikara would know that better. >> https://github.com/googleapis/java-pubsublite-spark >> > > It should be supported. See instructions here under "Portable > (Java/Python/Go)": https://beam.apache.org/documentation/runners/spark/ > > >> >> >> -Daniel >> >> On Thu, Aug 4, 2022 at 7:48 PM Drew Forbes < >> [email protected]> wrote: >> >>> I've actually not used PyBeam, I just meant writing Beam code with >>> Python. Didn't realize there was a whole separate PyBeam package. >>> >> > Thanks for clarifying. > > Thanks, > Cham > > >> >>> I feel dumb asking, but basically we just couldn't get the import to >>> work. I upgraded to apache-beam 2.40.0 and tried to access >>> apache_beam.io.gcp.pubsublite.ReadFromPubSubLite through various means >>> (regular import, proto_api, something like .external., etc) within Python >>> and determined that there just wasn't anything to access. We could >>> definitely have been wrong about that but it wasn't clear how to move >>> forward so we just switched our focus to writing native Spark code to pull >>> from PubSub Lite >>> >>> On Thu, Aug 4, 2022 at 6:46 PM Chamikara Jayalath <[email protected]> >>> wrote: >>> >>>> I believe this should be fully working. I'm not familiar with PyBeam >>>> though. Is the execution mechanism the same as running a regular Beam >>>> pipeline ? Also, note that for multi-language, you need to use a portable >>>> Beam runner. >>>> >>>> +Daniel Collins <[email protected]> who implemented this. >>>> >>>> Thanks, >>>> Cham >>>> >>>> On Thu, Aug 4, 2022 at 11:24 AM Austin Bennett < >>>> [email protected]> wrote: >>>> >>>>> Hi Users/Devs, >>>>> >>>>> Drew, copied, reported having troubles with PubSub Lite: >>>>> >>>>> "we just weren’t able to get PubSub Lite working with PyBeam. It’s >>>>> been a few weeks since we last tried, but we were just trying to use >>>>> `apache_beam.io.gcp.pubsublite.ReadFromPubSubLite` (here >>>>> <https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.pubsublite.html> >>>>> ) in PyBeam and couldn’t get it to import so we just gave up. From the >>>>> looks of the repo we couldn’t tell if it was ever actually fully >>>>> implemented and published" >>>>> >>>>> I haven't used myself, and figured others might be able to >>>>> comment/share at least if any have had success using and/or at least >>>>> whether fully tested/implemented IO ( whether available via cross-language >>>>> or 'native' python ). >>>>> >>>>> Please share any thoughts here. >>>>> >>>>> Cheers, >>>>> Austin >>>>> >>>>>
