HI Nick
My queue configuration is like this
<queue>
<name>appannie-queue</name>
<rate>60/s</rate>
<bucket-size>100</bucket-size>
<max-concurrent-requests>50</max-concurrent-requests>
<retry-parameters>
<task-retry-limit>0</task-retry-limit>
</retry-parameters>
</queue>
But with this, the problem is maximum of the tasks are getting failed and
the number of records which I am getting in the big query is very less its
about 300000 Lakh instead of 2400000 lakh. Not able to figure out how to
resolve this.
On Monday, March 21, 2016 at 10:51:38 AM UTC+5:30, Nickolas Daskalou wrote:
>
> Hi Hemanth,
>
> Is your task queue set up to allow enough concurrency and/or execution
> rate?
>
> See: https://cloud.google.com/appengine/docs/python/config/queue
>
> Nick
>
>
> On 21 March 2016 at 15:59, Hemanth Kumar <[email protected]
> <javascript:>> wrote:
>
>> HI Nick,
>>
>> This includes a call to the API getting the response and inserting
>> into big query. For the batch insert I am using TableRow only, but at a
>> time I can only form a batch of 3000 records, and then call it to the big
>> query. As I had written earlier, we have split our whole logic into a sets
>> of task. The total count of task is around 5000. And each task will give me
>> 3000 records. To complete all the task, how to reduce the time complexity,
>> its a biggest challenge for me.
>>
>> Earlier , before breaking the job into multiple task , I created only one
>> task, but to execute the whole task it was taking a long time because it
>> almost 5000 times we have to hit the rest api to get the response. So in
>> that case timeout exception was coming, to overcome this we splitted this
>> whole job into 5000 task. Now , its not throwing the timeout exception, but
>> the execution of all these task is taking a long time.
>>
>> Please suggest me how to overcome of this problem. It will be very useful
>> for me.
>>
>>
>>
>> On Saturday, March 19, 2016 at 12:56:50 AM UTC+5:30, Nick (Cloud Platform
>> Support) wrote:
>>>
>>> Hey Hemanth,
>>>
>>> There remain, after your post, some questions as to how you've
>>> implemented your system. The rate of inserts according to 47,000 / hour is
>>> approximately 13 / second. That seems very low - the maximum rate of
>>> inserts for streaming inserts is 100,000 rows per second, per table and
>>> 1,000,000 rows per second, per project.
>>> <https://cloud.google.com/bigquery/streaming-data-into-bigquery>
>>>
>>> Correct me if I'm wrong, but this starts to look like this number
>>> (47,000 in 1 hour) represents the overall time of the entire pipeline from
>>> API call to sending the BigQuery insert(), not simply the rate at which
>>> inserts could theoretically take place.
>>>
>>> Ultimately, the best way to insert BigQuery rows is not with isolated
>>> HTTP requests, which have a lot of overhead costs, but with batch inserts.
>>> It might be worth looking into ways that you could aggregate the records in
>>> a layer after the task pipeline but before BigQuery which would allow you
>>> to send big batched inserts.
>>>
>>> Let me know your thoughts on this, and best wishes,
>>>
>>> Nick
>>> Cloud Platform Community Support
>>>
>>> On Friday, March 18, 2016 at 8:46:59 AM UTC-4, Hemanth Kumar wrote:
>>>>
>>>>
>>>> HI ALL
>>>>
>>>> Urgently required some help.
>>>>
>>>> I am facing a slowness problem while writing the data to the big query.
>>>>
>>>> My problem is I have 5000 task , and each task is interacting with the
>>>> Rest Api. The rest api is giving me the JSON response. I am parsing the
>>>> JSON response, and the same response I have to write to big query.
>>>>
>>>> Every JSON response will give me around 3000 List of JSON Array.
>>>> If I calculate the total data which will get inserted in the google big
>>>> query is around 15000000.
>>>> To insert 47000 record its taking 1 hour. If I have to insert lakhs of
>>>> record of data , my performance is getting hit.
>>>>
>>>> Please give a good suggestion to improve the time complexity for this
>>>> problem. How I will resolve this using Google App Engine,
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> To post to this group, send email to [email protected]
>> <javascript:>.
>> Visit this group at https://groups.google.com/group/google-appengine.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/google-appengine/8475603f-a249-407a-b180-2e5f4b51ed54%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/google-appengine/8475603f-a249-407a-b180-2e5f4b51ed54%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/c2e1d973-aab7-490e-b4d2-4c22bd8af22d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
