Hi Ali, Sounds like a better thing to do. If the response gets too big, you'll probably want to store the results as a file or otherwise immediately process them in a bit of python code.
If you want to write a custom operator for this depends on how many times you wish to reuse this functionality, which depends on how many interfaces return larger responses and if it's appropriate to treat them in a standardized way for those cases. Another factor driving this decision is debuggability. If you have a standardized location on the cloud to store the file that you ingest, you can go back to that later to figure out which line or what went wrong. If you treat the response immediately, you probably don't have the file available and you should worry a bit more about general logging and fault handling. The easiest approach for a "one-off" is to use the Hook in a PythonOperator directly and treat the response there. Rgds, Gerard On Wed, Mar 8, 2017 at 5:18 PM, Ali Uz <[email protected]> wrote: > Hello, > > I want to use the HttpOperator to issue a GET request and pass a fairly > large response to another task. Is there any best way to handle this? I > think pushing a large object to xcom is not recommended. Should I just > write a custom operator that will fetch from the api and handle it as > necessary? > > Would love to hear some thoughts. > > Thanks! > > -Ali >
