The error is like process is already taken by some other request, like that its coming
On Mon, Mar 21, 2016 at 11:42 AM, Nickolas Daskalou <[email protected]> wrote: > Why are most of the tasks failing? What is the most common error when a > task fails? > > > On 21 March 2016 at 16:40, Hemanth Kumar <[email protected]> > wrote: > >> HI Nick >> >> My queue configuration is like this >> >> <queue> >> <name>appannie-queue</name> >> <rate>60/s</rate> >> <bucket-size>100</bucket-size> >> <max-concurrent-requests>50</max-concurrent-requests> >> <retry-parameters> >> <task-retry-limit>0</task-retry-limit> >> </retry-parameters> >> </queue> >> >> But with this, the problem is maximum of the tasks are getting failed and >> the number of records which I am getting in the big query is very less its >> about 300000 Lakh instead of 2400000 lakh. Not able to figure out how to >> resolve this. >> >> On Monday, March 21, 2016 at 10:51:38 AM UTC+5:30, Nickolas Daskalou >> wrote: >>> >>> Hi Hemanth, >>> >>> Is your task queue set up to allow enough concurrency and/or execution >>> rate? >>> >>> See: https://cloud.google.com/appengine/docs/python/config/queue >>> >>> Nick >>> >>> >>> On 21 March 2016 at 15:59, Hemanth Kumar <[email protected]> wrote: >>> >>>> HI Nick, >>>> >>>> This includes a call to the API getting the response and >>>> inserting into big query. For the batch insert I am using TableRow only, >>>> but at a time I can only form a batch of 3000 records, and then call it to >>>> the big query. As I had written earlier, we have split our whole logic into >>>> a sets of task. The total count of task is around 5000. And each task will >>>> give me 3000 records. To complete all the task, how to reduce the time >>>> complexity, its a biggest challenge for me. >>>> >>>> Earlier , before breaking the job into multiple task , I created only >>>> one task, but to execute the whole task it was taking a long time because >>>> it almost 5000 times we have to hit the rest api to get the response. So in >>>> that case timeout exception was coming, to overcome this we splitted this >>>> whole job into 5000 task. Now , its not throwing the timeout exception, but >>>> the execution of all these task is taking a long time. >>>> >>>> Please suggest me how to overcome of this problem. It will be very >>>> useful for me. >>>> >>>> >>>> >>>> On Saturday, March 19, 2016 at 12:56:50 AM UTC+5:30, Nick (Cloud >>>> Platform Support) wrote: >>>>> >>>>> Hey Hemanth, >>>>> >>>>> There remain, after your post, some questions as to how you've >>>>> implemented your system. The rate of inserts according to 47,000 / hour is >>>>> approximately 13 / second. That seems very low - the maximum rate of >>>>> inserts for streaming inserts is 100,000 rows per second, per table >>>>> and 1,000,000 rows per second, per project. >>>>> <https://cloud.google.com/bigquery/streaming-data-into-bigquery> >>>>> >>>>> Correct me if I'm wrong, but this starts to look like this number >>>>> (47,000 in 1 hour) represents the overall time of the entire pipeline from >>>>> API call to sending the BigQuery insert(), not simply the rate at which >>>>> inserts could theoretically take place. >>>>> >>>>> Ultimately, the best way to insert BigQuery rows is not with isolated >>>>> HTTP requests, which have a lot of overhead costs, but with batch inserts. >>>>> It might be worth looking into ways that you could aggregate the records >>>>> in >>>>> a layer after the task pipeline but before BigQuery which would allow you >>>>> to send big batched inserts. >>>>> >>>>> Let me know your thoughts on this, and best wishes, >>>>> >>>>> Nick >>>>> Cloud Platform Community Support >>>>> >>>>> On Friday, March 18, 2016 at 8:46:59 AM UTC-4, Hemanth Kumar wrote: >>>>>> >>>>>> >>>>>> HI ALL >>>>>> >>>>>> Urgently required some help. >>>>>> >>>>>> I am facing a slowness problem while writing the data to the big >>>>>> query. >>>>>> >>>>>> My problem is I have 5000 task , and each task is interacting with >>>>>> the Rest Api. The rest api is giving me the JSON response. I am parsing >>>>>> the >>>>>> JSON response, and the same response I have to write to big query. >>>>>> >>>>>> Every JSON response will give me around 3000 List of JSON Array. >>>>>> If I calculate the total data which will get inserted in the google >>>>>> big query is around 15000000. >>>>>> To insert 47000 record its taking 1 hour. If I have to insert lakhs >>>>>> of record of data , my performance is getting hit. >>>>>> >>>>>> Please give a good suggestion to improve the time complexity for this >>>>>> problem. How I will resolve this using Google App Engine, >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/google-appengine. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/google-appengine/8475603f-a249-407a-b180-2e5f4b51ed54%40googlegroups.com >>>> <https://groups.google.com/d/msgid/google-appengine/8475603f-a249-407a-b180-2e5f4b51ed54%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/google-appengine. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/google-appengine/c2e1d973-aab7-490e-b4d2-4c22bd8af22d%40googlegroups.com >> <https://groups.google.com/d/msgid/google-appengine/c2e1d973-aab7-490e-b4d2-4c22bd8af22d%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Google App Engine" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/google-appengine/0tp7ZF0Mv1I/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/google-appengine. > To view this discussion on the web visit > https://groups.google.com/d/msgid/google-appengine/CAOj3zuDaD0EAXhAaycOE2A_qA_B6%3Dgjfd37FbyXa4vMQA5UATQ%40mail.gmail.com > <https://groups.google.com/d/msgid/google-appengine/CAOj3zuDaD0EAXhAaycOE2A_qA_B6%3Dgjfd37FbyXa4vMQA5UATQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CALOvf0Hhi7ccagwqvec4f1hPYd%3DFxeDFiXn5qHTsCtBgudo4Kg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
