[appengine-java] Re: Model to Optimize Queries
Thanks Jason, I realize that I wont be able to do transactions per say
across the entity groups, and that the tasks can be run multiple times. With
those two constraints in mind, I designed both of the tasks described above
(running any of the two multiple times wont have any affect due to the stage
variable/pendingSet, and they rely on transactions within the same entity
group). Can you please take a deeper look on what the tasks actually do and
will they achieve what I am trying to do here. Just want to make sure I am
not missing anything basic with this.
Thanks for bringing out Option 3 as the easy winner with 30s rule I wasn't
considering. Thanks for all your help in general as well!
On Tue, Sep 22, 2009 at 10:39 AM, Jason (Google) apija...@google.comwrote:
Hi Sam. Based on your design above, I believe you're modeling an unowned
relationship between articles and reviewers, which makes sense.
Unfortunately, this means that you cannot use transactions, so you'll have
to work around this. Also keep in mind that tasks may be executed multiple
times so you will need to include logic to ensure that, should a task be
executed a second time, it won't throw off your application's state.
Regarding the task queue scenarios you laid out, I prefer the third option,
although the first and second should work. With the first and second
options, you need to be aware of how many articles and reviewers you're
processing since tasks are subject to the same 30-second request limits that
normal requests are. Option 3 neatly avoids this problem, which is one
reason why I consider it preferable.
- Jason
On Sun, Sep 20, 2009 at 8:46 PM, Sam Walker am.sam.wal...@gmail.comwrote:
Yea, I dont need to query on it, that's why I thought maybe storing in a
HashMap would save me an extra query. Lets not go that way for now and
assume I make a new relationship:
I have another question, I was wondering if I can use TaskQueue to achieve
some sort of transactions across entities and give my reviewers points (to
rank them later on) based on their reviews.
So, once an Article is finalized, I want to give each reviewer some points
(based on some logic like first reviewer who approved gets the maximum
points and so on and so forth), I am thinking of having another persistent
variable in Article named stage, initialized to 0 and a pendingSet in
Reviewer
Once everyone approves, I can set stage to 1.
Task 1:
For all articles in stage 1:
For all reviewers in this article
in a transaction add the given article to pendingSet of the reviewer
Set article stage to 2
Task 2:
For all articles in stage 2:
For all reviewers in this article:
in a transaction, remove the article from the pendingSet and add
points to the reviewer
Set article stage to 3
First and foremost, you think this will work, right?
Secondly, which approach is the best:
1. Make one big task which executes both task1 followed by task 2, and
run it every five minutes or so.
2. Run both task 1 and task 2 every 5 minutes
3. Change the tasks so that they take the key of the article as input, and
kick off Task 1 with key = articleKey after setting the stage to 1, and Task
1 kicks off Task2 with key = articleKey after setting the stage to 2.
Please feel free to add more and advise which should be used when.
Thanks a bunch!
On Tue, Sep 8, 2009 at 12:03 PM, Jason (Google) apija...@google.comwrote:
How do you plan to use the HashMap? You won't be able to query on it, so
if that's a requirement, you'll want to look into adding another
relationship.
- Jason
On Fri, Sep 4, 2009 at 7:14 PM, Sam Walker am.sam.wal...@gmail.comwrote:
Thanks!
I am thinking of doing something like this:
class Article {
HashSetKey reviewers;
HashSetString tags;
int status; // pending, approved, declined - derived from reviewers'
statuses
HashMapKey, Review mapping; // Reviewer key to Review mapping to
prevent storing making another 1:n relationship
class Review {
int status; // pending, approved, declined
String notes;
... // anything else
}
}
Do you think this will work? It doesn't get me all the things I want,
but is still good enough I think (its kind of sad that we can't model this
common scenario effectively/efficiently).
Do you recommend doing this HashMap kind of a hacky thing or just make a
new Entity and a new relationship? Is it worth the saving?
On Sat, Aug 29, 2009 at 12:28 AM, leszek leszek.ptokar...@gmail.comwrote:
Let consider a different approach. Take into account that Article and
Reviewer are rather immutable data (you need adding new article but
not to change existing), why break this nice feature.
Consider several classes:
Article { Key , {tags} next attribuites }
Reviewer { Key, mail, ... next attributes }
ArticleNotReviewedYet { Key articleKey }
ArticleUnderReview { Key articleKey, Key reviewerKey, int
reviewResult }
ArticleReviewed {Key
[appengine-java] Re: Model to Optimize Queries
Yea, I dont need to query on it, that's why I thought maybe storing in a
HashMap would save me an extra query. Lets not go that way for now and
assume I make a new relationship:
I have another question, I was wondering if I can use TaskQueue to achieve
some sort of transactions across entities and give my reviewers points (to
rank them later on) based on their reviews.
So, once an Article is finalized, I want to give each reviewer some points
(based on some logic like first reviewer who approved gets the maximum
points and so on and so forth), I am thinking of having another persistent
variable in Article named stage, initialized to 0 and a pendingSet in
Reviewer
Once everyone approves, I can set stage to 1.
Task 1:
For all articles in stage 1:
For all reviewers in this article
in a transaction add the given article to pendingSet of the reviewer
Set article stage to 2
Task 2:
For all articles in stage 2:
For all reviewers in this article:
in a transaction, remove the article from the pendingSet and add points
to the reviewer
Set article stage to 3
First and foremost, you think this will work, right?
Secondly, which approach is the best:
1. Make one big task which executes both task1 followed by task 2, and run
it every five minutes or so.
2. Run both task 1 and task 2 every 5 minutes
3. Change the tasks so that they take the key of the article as input, and
kick off Task 1 with key = articleKey after setting the stage to 1, and Task
1 kicks off Task2 with key = articleKey after setting the stage to 2.
Please feel free to add more and advise which should be used when.
Thanks a bunch!
On Tue, Sep 8, 2009 at 12:03 PM, Jason (Google) apija...@google.com wrote:
How do you plan to use the HashMap? You won't be able to query on it, so if
that's a requirement, you'll want to look into adding another relationship.
- Jason
On Fri, Sep 4, 2009 at 7:14 PM, Sam Walker am.sam.wal...@gmail.comwrote:
Thanks!
I am thinking of doing something like this:
class Article {
HashSetKey reviewers;
HashSetString tags;
int status; // pending, approved, declined - derived from reviewers'
statuses
HashMapKey, Review mapping; // Reviewer key to Review mapping to
prevent storing making another 1:n relationship
class Review {
int status; // pending, approved, declined
String notes;
... // anything else
}
}
Do you think this will work? It doesn't get me all the things I want, but
is still good enough I think (its kind of sad that we can't model this
common scenario effectively/efficiently).
Do you recommend doing this HashMap kind of a hacky thing or just make a
new Entity and a new relationship? Is it worth the saving?
On Sat, Aug 29, 2009 at 12:28 AM, leszek leszek.ptokar...@gmail.comwrote:
Let consider a different approach. Take into account that Article and
Reviewer are rather immutable data (you need adding new article but
not to change existing), why break this nice feature.
Consider several classes:
Article { Key , {tags} next attribuites }
Reviewer { Key, mail, ... next attributes }
ArticleNotReviewedYet { Key articleKey }
ArticleUnderReview { Key articleKey, Key reviewerKey, int
reviewResult }
ArticleReviewed {Key articleKey, int totalReviewResult }
This way if you want:
- to know the status of the article : find article in the first table
and find (by looking up the articleKey only) in what table
(ArticleNotReviewed, ArticleUnderReview, ArticleReviewed) contains the
articleKey
- to add new article: add article to Article and entry to
ArticleNotReviewed
- to start review: remove article from ArticleNotReviewed and add an
entry in ArticleUnderReview
- to add next review result: add next entry to ArticleUnderReview
- to end up review: remove all articleKey entries in
ArticleUnderReview and create entry in ArticleReviewed
Of course, it needs much more elaboration, but may be it is worth
considering.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Re: To specify Key or use auto generation
In 1, another problem is that I may not know the Keys of A and B at the
point of creation. For example, lets say I am doing something like this:
class Person {
}
class Group {
@Persistent(mappedBy = group)
ArrayListMembership memberships;
}
class Membership {
Key person;
Group group;
}
Now I would do:
Group g = new Group();
while(...) {
Membership m = new Membership();
g.memberships.add(m);
}
pm.makePersistent(g);
Now, when I create membership, I dont have the Group key (I do have the
person key), I can't really use it to create a new derived key for
membership. Or am I missing something?
On Sun, Sep 6, 2009 at 11:04 PM, Sam Walker am.sam.wal...@gmail.com wrote:
Lets say there is a relationship between Entities A and B, and I want to
model it with Entity C.
Two ways:
1. Use concatenated Key of A and B for Key of C. Data model will ensure
there are no duplicates.
2. Use autogenerated Key and make sure you do not duplicate A B
relationship in code.
Which one would you choose? In 1, is a simple concatenation the best way to
go about it?
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Re: Model to Optimize Queries
Thanks!
I am thinking of doing something like this:
class Article {
HashSetKey reviewers;
HashSetString tags;
int status; // pending, approved, declined - derived from reviewers'
statuses
HashMapKey, Review mapping; // Reviewer key to Review mapping to prevent
storing making another 1:n relationship
class Review {
int status; // pending, approved, declined
String notes;
... // anything else
}
}
Do you think this will work? It doesn't get me all the things I want, but is
still good enough I think (its kind of sad that we can't model this common
scenario effectively/efficiently).
Do you recommend doing this HashMap kind of a hacky thing or just make a new
Entity and a new relationship? Is it worth the saving?
On Sat, Aug 29, 2009 at 12:28 AM, leszek leszek.ptokar...@gmail.com wrote:
Let consider a different approach. Take into account that Article and
Reviewer are rather immutable data (you need adding new article but
not to change existing), why break this nice feature.
Consider several classes:
Article { Key , {tags} next attribuites }
Reviewer { Key, mail, ... next attributes }
ArticleNotReviewedYet { Key articleKey }
ArticleUnderReview { Key articleKey, Key reviewerKey, int
reviewResult }
ArticleReviewed {Key articleKey, int totalReviewResult }
This way if you want:
- to know the status of the article : find article in the first table
and find (by looking up the articleKey only) in what table
(ArticleNotReviewed, ArticleUnderReview, ArticleReviewed) contains the
articleKey
- to add new article: add article to Article and entry to
ArticleNotReviewed
- to start review: remove article from ArticleNotReviewed and add an
entry in ArticleUnderReview
- to add next review result: add next entry to ArticleUnderReview
- to end up review: remove all articleKey entries in
ArticleUnderReview and create entry in ArticleReviewed
Of course, it needs much more elaboration, but may be it is worth
considering.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Re: Model to Optimize Queries
Yes, it is a key. I just wanted to make it obvious which Keys are involved.
I read the Embedded class as well, its not what I want here I think.
How would you model this scenario then? Both models I discussed in my
original post have issues.
On Thu, Aug 27, 2009 at 11:22 AM, Jason (Google) apija...@google.comwrote:
JDO supports syntax like article.status but App Engine's datastore doesn't
support joins so you'll have to make article an embedded object in order to
use the query as you have it below:
http://code.google.com/appengine/docs/java/datastore/dataclasses.html#Embedded_Classes
Also, how have you defined reviewerParam? Is it a Key object or are you
actually passing in the Reviewer object? The latter won't work, but I'll
follow up about the former (querying on a Key).
- Jason
On Sun, Aug 23, 2009 at 11:18 PM, Sam Walker am.sam.wal...@gmail.comwrote:
Also, I get this error: *Can only filter by properties of a sub-object if
the sub-object is embedded.* when I tried to access article while setting
a fitler: query.setFilter(reviewer == reviewerParam article.status =
articleStatusParam);
What am I missing?
On Sat, Aug 22, 2009 at 6:02 PM, Sam Walker am.sam.wal...@gmail.comwrote:
In the second model, how will I find all Articles being reviewed by A and
B?
The only way I can think of is adding another derived field in Article:
Article {
HashSetReview reviews;
HashSetReviewer reviewers; // keys of Reviewers to help the query
find all articles reviewed by A and B
HashSetString tags;
int status; // derived from all Reviews' statuses.
}
Review {
Article article;
Reviewer reviewer;
int status;
}
Now I should be able to do reviewers.contains(A.key) and
reviewers.contains(B.key). Is that the best way?
On Sat, Aug 22, 2009 at 2:02 PM, Sam Walker am.sam.wal...@gmail.comwrote:
Oh sweet, I didn't know that.
That's awesome! Thanks for the quick reply.
On Sat, Aug 22, 2009 at 1:51 PM, datanucleus
andy_jeffer...@yahoo.comwrote:
I dont think I can do sth like (I can't access article.tags,
article.status
as far as I know, correct?):
Of course you can. JDO spec defines JDOQL, using Java syntax.
query.setFilter(reviewer == reviewerParam status == statusParam
article.tags == tagParam article.status = articleStatusParam);
What is tagParam ? an element of the hashSet?
You can't do Collection == element in Java so you can't in JDOQL.
You could do article.tags.contains(tagParam)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Re: Model to Optimize Queries
Any ideas, anyone?
On Sun, Aug 23, 2009 at 11:18 PM, Sam Walker am.sam.wal...@gmail.comwrote:
Also, I get this error: *Can only filter by properties of a sub-object if
the sub-object is embedded.* when I tried to access article while setting
a fitler: query.setFilter(reviewer == reviewerParam article.status =
articleStatusParam);
What am I missing?
On Sat, Aug 22, 2009 at 6:02 PM, Sam Walker am.sam.wal...@gmail.comwrote:
In the second model, how will I find all Articles being reviewed by A and
B?
The only way I can think of is adding another derived field in Article:
Article {
HashSetReview reviews;
HashSetReviewer reviewers; // keys of Reviewers to help the query
find all articles reviewed by A and B
HashSetString tags;
int status; // derived from all Reviews' statuses.
}
Review {
Article article;
Reviewer reviewer;
int status;
}
Now I should be able to do reviewers.contains(A.key) and
reviewers.contains(B.key). Is that the best way?
On Sat, Aug 22, 2009 at 2:02 PM, Sam Walker am.sam.wal...@gmail.comwrote:
Oh sweet, I didn't know that.
That's awesome! Thanks for the quick reply.
On Sat, Aug 22, 2009 at 1:51 PM, datanucleus
andy_jeffer...@yahoo.comwrote:
I dont think I can do sth like (I can't access article.tags,
article.status
as far as I know, correct?):
Of course you can. JDO spec defines JDOQL, using Java syntax.
query.setFilter(reviewer == reviewerParam status == statusParam
article.tags == tagParam article.status = articleStatusParam);
What is tagParam ? an element of the hashSet?
You can't do Collection == element in Java so you can't in JDOQL.
You could do article.tags.contains(tagParam)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Re: Model to Optimize Queries
Also, I get this error: *Can only filter by properties of a sub-object if
the sub-object is embedded.* when I tried to access article while setting a
fitler: query.setFilter(reviewer == reviewerParam article.status =
articleStatusParam);
What am I missing?
On Sat, Aug 22, 2009 at 6:02 PM, Sam Walker am.sam.wal...@gmail.com wrote:
In the second model, how will I find all Articles being reviewed by A and
B?
The only way I can think of is adding another derived field in Article:
Article {
HashSetReview reviews;
HashSetReviewer reviewers; // keys of Reviewers to help the query find
all articles reviewed by A and B
HashSetString tags;
int status; // derived from all Reviews' statuses.
}
Review {
Article article;
Reviewer reviewer;
int status;
}
Now I should be able to do reviewers.contains(A.key) and
reviewers.contains(B.key). Is that the best way?
On Sat, Aug 22, 2009 at 2:02 PM, Sam Walker am.sam.wal...@gmail.comwrote:
Oh sweet, I didn't know that.
That's awesome! Thanks for the quick reply.
On Sat, Aug 22, 2009 at 1:51 PM, datanucleus andy_jeffer...@yahoo.comwrote:
I dont think I can do sth like (I can't access article.tags,
article.status
as far as I know, correct?):
Of course you can. JDO spec defines JDOQL, using Java syntax.
query.setFilter(reviewer == reviewerParam status == statusParam
article.tags == tagParam article.status = articleStatusParam);
What is tagParam ? an element of the hashSet?
You can't do Collection == element in Java so you can't in JDOQL.
You could do article.tags.contains(tagParam)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Model to Optimize Queries
Model: Article can have multiple tags and multiple reviewers
Queries I want to do:
1. Find all articles with a particular tag which I have to review
2. Find all articles that are being reviewed by me and someone else
Model I am thinking:
Article {
HashSetReviewer reviewers;
HashSetString tags;
}
Now the twist is that reviewers can approve or disapprove the article for
publishing, so I want to show a dashboard where I would do queries like:
1. Articles I still need to review.
2. Articles which you have approved, but haven't been approved by everyone
else (we could be waiting on someone's response or someone could have
declined (you may want to take some action, talk to the other reviewer)).
and I was thinking that there should also be a way to see
3. Articles that I have declined.
4. Articles that have been approved.
I was initially thinking of keeping a HashMapKey, int status; kind of a
thing, but then I would have to go through every Article, that can't be
good. I also thought of doing this:
Article {
HashSetReview reviews;
HashSetString tags;
int status; // derived from all Reviews' statuses.
}
Review {
Article article;
Reviewer reviewer;
int status;
}
I dont think I can do sth like (I can't access article.tags, article.status
as far as I know, correct?):
final Query query = pm.newQuery(Review.class);
query.setFilter(reviewer == reviewerParam status == statusParam
article.tags == tagParam article.status = articleStatusParam);
So, whats the best way to model this scenario, where should I compromise?
Thanks!
P.S. Also wondering how do people search across all these entities so fast?
Whats the catch?
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
[appengine-java] Re: Model to Optimize Queries
Oh sweet, I didn't know that. That's awesome! Thanks for the quick reply. On Sat, Aug 22, 2009 at 1:51 PM, datanucleus andy_jeffer...@yahoo.comwrote: I dont think I can do sth like (I can't access article.tags, article.status as far as I know, correct?): Of course you can. JDO spec defines JDOQL, using Java syntax. query.setFilter(reviewer == reviewerParam status == statusParam article.tags == tagParam article.status = articleStatusParam); What is tagParam ? an element of the hashSet? You can't do Collection == element in Java so you can't in JDOQL. You could do article.tags.contains(tagParam) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en -~--~~~~--~~--~--~---
[appengine-java] Re: Model to Optimize Queries
In the second model, how will I find all Articles being reviewed by A and B?
The only way I can think of is adding another derived field in Article:
Article {
HashSetReview reviews;
HashSetReviewer reviewers; // keys of Reviewers to help the query find
all articles reviewed by A and B
HashSetString tags;
int status; // derived from all Reviews' statuses.
}
Review {
Article article;
Reviewer reviewer;
int status;
}
Now I should be able to do reviewers.contains(A.key) and
reviewers.contains(B.key). Is that the best way?
On Sat, Aug 22, 2009 at 2:02 PM, Sam Walker am.sam.wal...@gmail.com wrote:
Oh sweet, I didn't know that.
That's awesome! Thanks for the quick reply.
On Sat, Aug 22, 2009 at 1:51 PM, datanucleus andy_jeffer...@yahoo.comwrote:
I dont think I can do sth like (I can't access article.tags,
article.status
as far as I know, correct?):
Of course you can. JDO spec defines JDOQL, using Java syntax.
query.setFilter(reviewer == reviewerParam status == statusParam
article.tags == tagParam article.status = articleStatusParam);
What is tagParam ? an element of the hashSet?
You can't do Collection == element in Java so you can't in JDOQL.
You could do article.tags.contains(tagParam)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~--~~~~--~~--~--~---
