Probably you should be returning "response" from the "get_responses()" function ?
On Mon, Jun 17, 2019 at 4:28 PM Anjana Pydi <[email protected]> wrote: > Hi Lukasz, > > Thanks for reply ! I tried in similar way but ParDo2 is not performing > given function on each individual element, its sending whole list. > > Below is the sample code snippet (this uses Map) : > > def get_responses(id): > get_req = requests.get(url_get) > output = get_req.json() > response = output['result'] #Which will return a list of dictionaries > > #Below method should take each dictionary obtained from get_responses > method and post it to API endpoint > def send_to_api(input): > post_req = requests.post(url_post, json.dumps(input)) > if post_req.ok: > print 'success' > > with beam.Pipeline(options=PipelineOptions()) as p: > elements = (p | beam.Create(list of IDs) > | 'get response' >> beam.Map(lambda id: get_responses(id)) > | 'post to API' >> beam.Map(lambda x: send_to_api(x)) > > Please let me know in case if any information needed. > > Thanks, > Anjana > > > ------------------------------ > *From:* Lukasz Cwik [[email protected]] > *Sent:* Monday, June 17, 2019 3:58 PM > *To:* user > *Subject:* Re: Can we do pardo inside a pardo? > > Typically you would apply your first ParDo getting back a PCollection and > then apply your second ParDo to the return PCollection. You can get a lot > more details in the programming guide[1]. > > For example: > PCollection<T> input = ... > input.apply("ParDo1", ParDo.of(myDoFn1)).apply("ParDo2", > ParDo.of(myDoFn2)); > > In Python it would be > input = ... > input | "ParDo1" >> beam.ParDo(myDoFn1) | "ParDo2" >> beam.ParDo(myDoFn2) > > 1: > https://beam.apache.org/documentation/programming-guide/#applying-transforms > > On Mon, Jun 17, 2019 at 3:18 PM Anjana Pydi <[email protected]> > wrote: > >> Hi, >> >> I want to do following steps: >> >> 1. For a list of ids, Do a Get request on an API which will give back a >> list of dictionaries as response. [ParDo1] >> 2. For each of dictionary obtained in Step1 , make some changes and post >> as JSON to API endpoint. [ParDo2] >> >> Step1 is working but not sure how to use second ParDo. Please let me know >> your suggestions. >> >> Thanks, >> Anjana >> ----------------------------------------------------------------------------------------------------------------------- >> The information contained in this communication is intended solely for the >> use of the individual or entity to whom it is addressed and others >> authorized to receive it. It may contain confidential or legally privileged >> information. If you are not the intended recipient you are hereby notified >> that any disclosure, copying, distribution or taking any action in reliance >> on the contents of this information is strictly prohibited and may be >> unlawful. If you are not the intended recipient, please notify us >> immediately by responding to this email and then delete it from your >> system. Bahwan Cybertek is neither liable for the proper and complete >> transmission of the information contained in this communication nor for any >> delay in its receipt. >> > ----------------------------------------------------------------------------------------------------------------------- > The information contained in this communication is intended solely for the > use of the individual or entity to whom it is addressed and others > authorized to receive it. It may contain confidential or legally privileged > information. If you are not the intended recipient you are hereby notified > that any disclosure, copying, distribution or taking any action in reliance > on the contents of this information is strictly prohibited and may be > unlawful. If you are not the intended recipient, please notify us > immediately by responding to this email and then delete it from your > system. Bahwan Cybertek is neither liable for the proper and complete > transmission of the information contained in this communication nor for any > delay in its receipt. >
