Re: [google-appengine] Re: Cast some light on Task Queue execution. I am confused

['Alex Martelli' via Google App Engine]

On Sun, Dec 27, 2015 at 9:32 PM, Naresh Pokuri  wrote:

> Thanks for the reply. When will be bucket refilled? what parameters impact
> bucket filling?
>

One token is added to the bucket every 1/rate -- so if your rate is 60/h,
one token will be added every minute. However, no tokens are added if the
bucket is already full (i.e it contains bucket-size tokens; default
bucket-size is 5 unless explicitly specified).

No other parameter beyond rate and bucket-size constrain bucket filling.

A queued task will grab a token and execute IF there's at least a token
available in the bucket AND this doesn't push the number of tasks
concurrently executing on that queue beyond the max-concurrent-requests
parameter (default for the latter, 1000, unless explicitly specified).
However, max-concurrent-requests constrains task execution, NOT bucket
filling.

I think wikipedia, at https://en.wikipedia.org/wiki/Token_bucket , does a
good job of presenting the general idea of token-bucket algorithms, beyond
the specific use of it in app engine task queues in push mode.


Alex



>
> I have the same configuration in my project for a queue and I have a cron
> job which adds tasks per every five minutes to that queue. On google
> console I see one task getting executed in a minute. If I click on 'Run
> Now' on console all are getting executed in 5 sec. What makes my queue
> execute a task per minute. Am I missing something here?
>
> On Friday, 25 December 2015 02:22:04 UTC+5:30, Zeehad (Cloud Platform
> Support) wrote:
>>
>> The 5 tasks in the ‘push-task’ queue will be completed after 5 seconds
>> given that they each have 1 sec latency.
>>
>> 1. 120 means the bucket will have 120 tokens
>> initially.
>>
>> 2.  60/h will mean a maximum of 60 tasks will be processed
>> per hour.
>>
>> For example, consider the following:
>>
>>
>>-
>>
>>Your above mentioned configurations
>>-
>>
>>100 tasks in the queue
>>-
>>
>>1 second latency each
>>
>>
>> Due to max-concurrent-request being 1 and rate only allowing 60 tasks per
>> hour, 60 tasks will complete in the first 60 seconds. Then no more tasks
>> will be processed for the next 59 minutes of that hour. After the hour is
>> over, the remaining 40 tasks will be processed in 40 seconds.
>>
>> Hope that helps. Cheers!
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/7d587334-72d9-42af-9fa7-616868838ac9%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAE46Be9Up_4No08BQO7ikS%3Dzeb43_SOBmJ2FzmXd8CRX1pYSag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Fwd: Us congress hearing of maan alsaan Money laundry قضية الكونغجرس لغسيل الأموال للمليادير معن الصانع

[sam a]

US Congressional Hearing of

 Saudi billionaire" maan  Al Sanea "

 and Money Laundering

with bank of America



With Arabic Subtitles





http://www.youtube.com/watch?v=mIBNnQvhU8s






*موقع اليوتيوب الذي عرض فيديوهات جلسة استماع الكونجرس الأمريكي *

* لمتابعة نشاطات غسل الأموال ونشاطات*



*السعودي معن عبدالواحد الصانع*

*مالك مستشفى  وشركة سعد  ومدارس سعد بالمنطقة الشرقية** بالسعودية * * ورئيس
مجلس ادارة بنك اوال البحريني*



 *وتعليق محطة سي ان بي سي التلفزيونية*



*مترجم باللغة العربية*


http://www.youtube.com/watch?v=mIBNnQvhU8s

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CA%2BHMVP1z7vMW6XRmWiOrKh2Lsar3fTwXahgGQg6FhPz3r%2Bzm%3DA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Data Modeling help for simple messaging

[timh]

To be honest.

I would try a niave implementation and deal with how you will 
fetch/send/receive information and how you plan to implement your workflows.
Then profile the application and then start looking at how you might 
optimise things.

Doing things like "I then thought of creating an entity to store all the 
sent/received messages in a string for a user inside of a single entity" 
means you can't scale the number of messages without dealing with 
overflows, and will people read old messages.  If not then retrieving that 
single growing entity may cost you more than the writes (in terms of time, 
cache hit ratios etc)).  Trying to optimise things before you have working 
code is possibly not the best way to approach development.


T

On Tuesday, December 29, 2015 at 12:29:48 PM UTC+8, Susan Lin wrote:
>
> I have a very simple requirement. I have an application where users log 
> in, add/remove/view friends, and can send/receive (also view the message 
> you sent and received) a message (message is more like an email, not a real 
> time chat) from a friend. When they receive the message, the application 
> needs to mark the message as 'read'. The messages must also come ordered by 
> newest to oldest.
>
> My thoughts:
>
> KINDS:
>
>
> USER
> username  (ID)
> password 
> friends  (will contain a max of 100 
> friends).
>
> MESSAGE
> msg_id  (ID)
> from  (indexed)
> to  (indexed)
> timestamp (indexed)
> msg
> hasRead
>
>
> 1) User would login using their username and password
> 2) User could get friends by getting their USER entity based on their 
> username
> 3) User could add/remove friends by getting the entity of user1 and user2 
> and either adding or removing friend via transaction to make sure they are 
> consistent,
> 4) User could get all the message they have sent by using indexing the the 
> 'from' attribute (limit of 10 message per request). The same could be done 
> to view all the messages they have received by using the 'to' attribute. 
> When the message has been seen for the first time I would go to the message 
> (1 get) and update the entity (1 write + (4 writes x 3) = 13 writes to 
> update the entity).
>  
> My major concern - If a user gets 10 messages, this will require 10 get 
> requests plus if all 10 messages are new I will need to update each entity 
> and if all of them are new that is (10 x 14) 140 writes. Then if I get 
> another 10 message for this user the same process and this could all add up 
> very quickly.
>
>
> I then thought of creating an entity to store all the sent/received 
> messages in a string for a user inside of a single entity:
>
> MESSAGE
> user_entitynum1  (ID)
> messages  msg2_touser8_read_timestamp, msg3_touser5_unread_timestamp, etc>
>
> This way I could store all the message (under 1mb) inside of this one 
> entity, but I would have to keep track of which entity each user is at (if 
> the user exceeds the 1mb limit I will have to create a new entity).  This 
> proved to also not be too efficient because if I send a message to perform 
> 2 gets to see which message entity I am currently at and which message 
> entity they are currently at. From there I must now use another 2 reads to 
> get those entities and another 4 writes to update them both (considering I 
> do not need to create another entity if it is full). 
>
>
> I was looking for any ideas to accomplish what I need in a more efficient 
> way. I have nothing implemented yet so I  am open to ANY ideas. My main 
> concern is for this to be efficient,
>
> Cheers!
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/e5e1e5f6-3875-4590-b26e-fdfeff728253%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Data Modeling help for simple messaging

[Susan Lin]

I have a very simple requirement. I have an application where users log in, 
add/remove/view friends, and can send/receive (also view the message you 
sent and received) a message (message is more like an email, not a real 
time chat) from a friend. When they receive the message, the application 
needs to mark the message as 'read'. The messages must also come ordered by 
newest to oldest.

My thoughts:

KINDS:


USER
username  (ID)
password 
friends  (will contain a max of 100 
friends).

MESSAGE
msg_id  (ID)
from  (indexed)
to  (indexed)
timestamp (indexed)
msg
hasRead


1) User would login using their username and password
2) User could get friends by getting their USER entity based on their 
username
3) User could add/remove friends by getting the entity of user1 and user2 
and either adding or removing friend via transaction to make sure they are 
consistent,
4) User could get all the message they have sent by using indexing the the 
'from' attribute (limit of 10 message per request). The same could be done 
to view all the messages they have received by using the 'to' attribute. 
When the message has been seen for the first time I would go to the message 
(1 get) and update the entity (1 write + (4 writes x 3) = 13 writes to 
update the entity).
 
My major concern - If a user gets 10 messages, this will require 10 get 
requests plus if all 10 messages are new I will need to update each entity 
and if all of them are new that is (10 x 14) 140 writes. Then if I get 
another 10 message for this user the same process and this could all add up 
very quickly.


I then thought of creating an entity to store all the sent/received 
messages in a string for a user inside of a single entity:

MESSAGE
user_entitynum1  (ID)
messages 

This way I could store all the message (under 1mb) inside of this one 
entity, but I would have to keep track of which entity each user is at (if 
the user exceeds the 1mb limit I will have to create a new entity).  This 
proved to also not be too efficient because if I send a message to perform 
2 gets to see which message entity I am currently at and which message 
entity they are currently at. From there I must now use another 2 reads to 
get those entities and another 4 writes to update them both (considering I 
do not need to create another entity if it is full). 


I was looking for any ideas to accomplish what I need in a more efficient 
way. I have nothing implemented yet so I  am open to ANY ideas. My main 
concern is for this to be efficient,

Cheers!


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/980c6d49-83dc-4fac-aaea-1521cf14b52e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: End points giving error when used with objectify ref<> and key<> fields

[Sheena Arora]

I posted this question on stackoverflow, but didn't get any answers. So 
thought about posting over here.

Anyway thanks 

On Sunday, December 27, 2015 at 5:02:33 AM UTC+5:30, Adam (Cloud Platform 
Support) wrote:
>
> This seems like a question that would be better posted to Stack Overflow, 
> rather than a discussion post. The google-appengine group is intended more 
> for general discussion rather than 1-on-1 support assistance. Please see 
> the main Community Support page 
>  for a list of the tags 
> we monitor.
>
> This is an issue that comes up commonly, because Endpoints doesn't support 
> arbitrary objects, however there are ways to work around this. Some 
> discussions and possible solutions to your case can be found here 
> 
>  
> and here 
> 
> . 
>
> On Saturday, December 26, 2015 at 2:42:17 AM UTC-5, Sheena Arora wrote:
>>
>> Hi,
>>
>> I am trying to create enpoints (backend APIs) for a java class using 
>> objectify to connect with GAE. When I try to access the api using api 
>> explorer , I get the error "java.lang.IllegalArgumentException: 
>> Parameterized type com.googlecode.objectify.Ref<> not supported."
>>
>> I have an entity which is having fields as ref<> of other entities.At 
>> this point, I can not change the design of the class.
>>
>> Thanks
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/5d2f547b-bf88-41d3-9690-839c3bcbeefa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] datastore timeouts?

[Adam Sah]

anyone else seeing increased datastore timeouts?  I've confirmed that 
traffic levels are normal i.e. not getting DOS'd...

E 2015-12-28 15:19:09.728 error:Timeout('The datastore operation timed out, 
or the data was temporarily unavailable.',)
...
E 2015-12-28 15:08:31.447 error:TransactionFailedError('too much contention 
on these datastore entities. please try again.',)

adam

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/f458c5e9-ae4e-4f5b-9d0f-04d792d79646%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.