RE: Can we do pardo inside a pardo?

[Anjana Pydi]

Hi Lukasz,

Thanks for reply ! I tried in similar way but ParDo2 is not performing given 
function on each individual element, its sending whole list.

Below is the sample code snippet (this uses Map) :

def get_responses(id):
    get_req = requests.get(url_get)
    output = get_req.json()
    response = output['result'] #Which will return a list of dictionaries

#Below method should take each dictionary obtained from get_responses method 
and post it to API endpoint
def send_to_api(input):
    post_req = requests.post(url_post, json.dumps(input))
    if post_req.ok:
        print 'success'

with beam.Pipeline(options=PipelineOptions()) as p:
                elements = (p | beam.Create(list of IDs)
                | 'get response' >> beam.Map(lambda id: get_responses(id))
                | 'post to API' >> beam.Map(lambda x: send_to_api(x))

Please let me know in case if any information needed.

Thanks,
Anjana


________________________________
From: Lukasz Cwik [lc...@google.com]
Sent: Monday, June 17, 2019 3:58 PM
To: user
Subject: Re: Can we do pardo inside a pardo?

Typically you would apply your first ParDo getting back a PCollection and then 
apply your second ParDo to the return PCollection. You can get a lot more 
details in the programming guide[1].

For example:
PCollection<T> input = ...
input.apply("ParDo1", ParDo.of(myDoFn1)).apply("ParDo2", ParDo.of(myDoFn2));

In Python it would be
input = ...
input | "ParDo1" >> beam.ParDo(myDoFn1) | "ParDo2" >> beam.ParDo(myDoFn2)

1: https://beam.apache.org/documentation/programming-guide/#applying-transforms

On Mon, Jun 17, 2019 at 3:18 PM Anjana Pydi 
<anjan...@bahwancybertek.com<mailto:anjan...@bahwancybertek.com>> wrote:
Hi,

I want to do following steps:

1. For a list of ids, Do a Get request on an API which will give back a list of 
dictionaries as response. [ParDo1]
2. For each of dictionary obtained in Step1 , make some changes and post as 
JSON to API endpoint. [ParDo2]

Step1 is working but not sure how to use second ParDo. Please let me know your 
suggestions.

Thanks,
Anjana
-----------------------------------------------------------------------------------------------------------------------
 The information contained in this communication is intended solely for the use 
of the individual or entity to whom it is addressed and others authorized to 
receive it. It may contain confidential or legally privileged information. If 
you are not the intended recipient you are hereby notified that any disclosure, 
copying, distribution or taking any action in reliance on the contents of this 
information is strictly prohibited and may be unlawful. If you are not the 
intended recipient, please notify us immediately by responding to this email 
and then delete it from your system. Bahwan Cybertek is neither liable for the 
proper and complete transmission of the information contained in this 
communication nor for any delay in its receipt.
-----------------------------------------------------------------------------------------------------------------------
 The information contained in this communication is intended solely for the use 
of the individual or entity to whom it is addressed and others authorized to 
receive it. It may contain confidential or legally privileged information. If 
you are not the intended recipient you are hereby notified that any disclosure, 
copying, distribution or taking any action in reliance on the contents of this 
information is strictly prohibited and may be unlawful. If you are not the 
intended recipient, please notify us immediately by responding to this email 
and then delete it from your system. Bahwan Cybertek is neither liable for the 
proper and complete transmission of the information contained in this 
communication nor for any delay in its receipt.