On Thu, Oct 30, 2014 at 2:43 PM, Jonás Mora <[email protected]> wrote:
> The ETL task is done in a background job that connects to the ftp server, > iterates through the files, and process each one of them. The process > consists in downloading the file, parsing it and transforming it to a > bigquery supported format. I'm using the streaming insert API to load the > data into bigquery but I read that this is not the best alternative for > batch processing. I don't understand how to do it otherwise. I would really > appreciate some help here. > Source: > https://cloud.google.com/developers/articles/bigquery-in-practice/#h.757bq5t4k2m8 > Assuming you've transformed the source data files into CSV, you can use the demo code at this link to batch-load the file data into BigQuery: https://cloud.google.com/bigquery/loading-data-into-bigquery#loaddatagcs . Note that *[sourceCSV]* is a URI to the CSV file hosted on Cloud Storage. But if you're not experiencing any problems with streaming insert, there's nothing wrong with continuing with your existing design. On Thu, Oct 30, 2014 at 2:43 PM, Jonás Mora <[email protected]> wrote: > Another issue I ran into is that when running multiple background threads > with ndb.multi_put I get "Process terminated because the backend was > stopped". > Is that the exact/only text that was printed out into logging? It's more than likely you're hitting some memory limits or throughput throttles. Try lowering the number of threads/increasing the number of instances running. Also try reducing the size and number of puts you're doing within each multi_put call. Can you give a general description of how much data is added during each multi_put call, and the general design of the entity model? ----------------- -Vinny P Technology & Media Consultant Chicago, IL App Engine Code Samples: http://www.learntogoogleit.com -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
