You want Beam.FlataMap(get_response) here so that the output PCollection contains all the elements of all the lists returned by get_response rather than the lists themselves.
On Tue, Jun 18, 2019, 3:27 AM Chamikara Jayalath <[email protected]> wrote: > > Probably you should be returning "response" from the "get_responses()" > function ? > > On Mon, Jun 17, 2019 at 4:28 PM Anjana Pydi <[email protected]> > wrote: > >> Hi Lukasz, >> >> Thanks for reply ! I tried in similar way but ParDo2 is not performing >> given function on each individual element, its sending whole list. >> >> Below is the sample code snippet (this uses Map) : >> >> def get_responses(id): >> get_req = requests.get(url_get) >> output = get_req.json() >> response = output['result'] #Which will return a list of dictionaries >> >> #Below method should take each dictionary obtained from get_responses >> method and post it to API endpoint >> def send_to_api(input): >> post_req = requests.post(url_post, json.dumps(input)) >> if post_req.ok: >> print 'success' >> >> with beam.Pipeline(options=PipelineOptions()) as p: >> elements = (p | beam.Create(list of IDs) >> | 'get response' >> beam.Map(lambda id: get_responses(id)) >> | 'post to API' >> beam.Map(lambda x: send_to_api(x)) >> >> Please let me know in case if any information needed. >> >> Thanks, >> Anjana >> >> >> ------------------------------ >> *From:* Lukasz Cwik [[email protected]] >> *Sent:* Monday, June 17, 2019 3:58 PM >> *To:* user >> *Subject:* Re: Can we do pardo inside a pardo? >> >> Typically you would apply your first ParDo getting back a PCollection and >> then apply your second ParDo to the return PCollection. You can get a lot >> more details in the programming guide[1]. >> >> For example: >> PCollection<T> input = ... >> input.apply("ParDo1", ParDo.of(myDoFn1)).apply("ParDo2", >> ParDo.of(myDoFn2)); >> >> In Python it would be >> input = ... >> input | "ParDo1" >> beam.ParDo(myDoFn1) | "ParDo2" >> beam.ParDo(myDoFn2) >> >> 1: >> https://beam.apache.org/documentation/programming-guide/#applying-transforms >> >> On Mon, Jun 17, 2019 at 3:18 PM Anjana Pydi <[email protected]> >> wrote: >> >>> Hi, >>> >>> I want to do following steps: >>> >>> 1. For a list of ids, Do a Get request on an API which will give back a >>> list of dictionaries as response. [ParDo1] >>> 2. For each of dictionary obtained in Step1 , make some changes and post >>> as JSON to API endpoint. [ParDo2] >>> >>> Step1 is working but not sure how to use second ParDo. Please let me >>> know your suggestions. >>> >>> Thanks, >>> Anjana >>> ----------------------------------------------------------------------------------------------------------------------- >>> The information contained in this communication is intended solely for the >>> use of the individual or entity to whom it is addressed and others >>> authorized to receive it. It may contain confidential or legally privileged >>> information. If you are not the intended recipient you are hereby notified >>> that any disclosure, copying, distribution or taking any action in reliance >>> on the contents of this information is strictly prohibited and may be >>> unlawful. If you are not the intended recipient, please notify us >>> immediately by responding to this email and then delete it from your >>> system. Bahwan Cybertek is neither liable for the proper and complete >>> transmission of the information contained in this communication nor for any >>> delay in its receipt. >>> >> ----------------------------------------------------------------------------------------------------------------------- >> The information contained in this communication is intended solely for the >> use of the individual or entity to whom it is addressed and others >> authorized to receive it. It may contain confidential or legally privileged >> information. If you are not the intended recipient you are hereby notified >> that any disclosure, copying, distribution or taking any action in reliance >> on the contents of this information is strictly prohibited and may be >> unlawful. If you are not the intended recipient, please notify us >> immediately by responding to this email and then delete it from your >> system. Bahwan Cybertek is neither liable for the proper and complete >> transmission of the information contained in this communication nor for any >> delay in its receipt. >> >
