There is also Romain's filesystem PR that wraps vfs as a Beam filesystem . That would at least in theory allow to read from HTTP and FTP, no?
Of course this is different from RestIO because it won't have the full HTTP verbs semantics but if the goal is just to read from (GET), maybe we should try to make this PR advance no? https://github.com/apache/beam/pull/4803 On Wed, Apr 11, 2018 at 3:50 PM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Hi Daniel, > > I know JB has a started a RestIO, it is not yet complete and needs > some love but probably some opportunity for converge and collaboration > here > > Romain Manni-Bucau > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book > > > 2018-04-11 15:42 GMT+02:00 Daniel Salerno <salerno....@gmail.com>: >> Good morning, >> >> In my Big Data Google project we need to read batch data from the VTEX >> platform every 15 minutes and record json's return to our cloud storage >> datalake. >> It makes the data available through HTTP GET requests for its API: >> (https://documenter.getpostman.com/view/487146/vtex-oms-api/6tjSKqi) >> >> I have not found a good way to do this reading using standard Apache Beam >> components, my idea is to develop an HttpIO component using Source Sink to >> read the request and write a JSON file to Cloud Storage. After that another >> pipeline will process the file and write to BigQuery. >> >> Can you please help me with this matter? Is that the best way to go? Do you >> have any idea how to develop this new component? >> >> Thank you. >> >> Hugs, >> >> Daniel Salerno de Arruda