The way to do it is if you think of books and friends as streams.
This way they are both the same model and you only need one model/api to handle
both
A book i have read is a stream
and a friend is a stream.
All you now do is subscribe users to friends streams
and users to book streams or book tag streams. you can also map streams to
streams which then allows tags to include other tags - if you really want to
get
nifty.
So the tag - Horror is like a stream and n number of users can subscribe to its
stream. the stream can then have a path stream/books/horror which then can
return the activity of that stream and the same for a user use a path as the
stream name stream/user/martin - which of course would now be my activity
stream.
This is basically done using the model that you have been give so far.
#parent
class stream(db.Model):
#key_name=path
Count=db.IntegerProperty(default=0) #how many lists we have used - n
#child
class stream_index(db.Model):
subscribers = db.StringListProperty() - dont use your user name use a
stream path - like stream/user/martin
all you now do is create your stream and append your subscribers to it in the
list property
BUT this brings a whole new contention issue. As the solution above allows
streams to have millions of subscribers.If they all subscribe or unsubscribe
you
may have update issues on the same property - which is why we use sharded
counters for counts. To work around this you use a pipeline. 2010 IO did a
whole
1.5hr chat on this and its good stuff. If you watch it a few times its simple
to
grasp and Brets solution can be used for this proposal.
basically rather than append your users to your steam as you would normally
subscribers.append(user.name)
You append them to a work-line - basically a pipe line of work that is picked
off by a task queue and then batched for updation in your streams. the work is
then removed from the work line.
class work_line(db.Model):
work_index=db.StringProperty() - the work index is a key that depicts the
task that is fired and what stream that task is for
subscriber=db.StringProperty() #+martin or -martin
The above is a simple work line that simply stores the user name to be
subscribed + or unsubscribe -
This sounds all very complex and possibly it is - the good news as i am
building
a stream subscription service using a pipeline
view the latest 2010 IO PIPELINES and 2009 building scalable web apps - the
links are posted all over this group.
Regards
Martin Webb
The information contained in this email is confidential and may contain
proprietary information. It is meant solely for the intended recipient. Access
to this email by anyone else is unauthorised. If you are not the intended
recipient, any disclosure, copying, distribution or any action taken or omitted
in reliance on this, is prohibited and may be unlawful. No liability or
responsibility is accepted if information or data is, for whatever reason
corrupted or does not reach its intended recipient. No warranty is given that
this email is free of viruses. The views expressed in this email are, unless
otherwise stated, those of the author
________________________________
From: Jeff Schwartz <[email protected]>
To: [email protected]
Sent: Wed, 7 July, 2010 18:56:10
Subject: Re: [google-appengine] list-property, many to many, 5000
indexes...HELP!
I wouldn't put the list attribute in your book class due to serialization
issues
and exploding indexes. I'd put the list in a sub class of book and query the
subclass to return a list of entity keys on a match. As child keys contain
parent keys as well, I'd use the list of keys returned to grab the actual book
entities. You can refine this as much as you like.
Jeff
On Wed, Jul 7, 2010 at 1:58 AM, stefoid <[email protected]> wrote:
Hi I think I have a fair handle on how Datastore works, but I need to
>check, and I need some help with a design
>
>lets say I have:
>a very large list of BOOKS
>the BOOKS are tagged- "adventure+romance", "historical+drama",
>"animation+sci-fi+comedy", etc...
>and I have a very large list of USERS.
>USERS can read a very large number of BOOKS
>And I, as a USER, can have a very large number of USERS who are my
>friend.
>
>What I want , in english, is
>; Return a list of all the BOOKS tagged with "sci-fi" and "romance"
>that have been read by USERS who are my friend, sorted by Most
>Frequently Read.
>
>Now, I know I can model tags with a ListProperty, so that filtering is
>easy = "WHERE tag AND tag AND tag..."
>And I know that I can sort Books by Most Frequently Read quite easily
>
>But due to the large number of BOOKS that could be read by a USER,
>and the large number of USERS who could be my friend, I cant
>practically model BOOKS READ BY USERS and USERS WHO ARE MY FRIEND as
>ListProperties...
>
>...Because I would blow the 5000 index limit per entity, because each
>value of ListPropery generates its own index entry. Is that correct?
>
>I cant find a way to handle this without resorting to filtering in
>memory. Am I dumb, or is that just the way it is?
>
>My solution is:
>
>BOOK:
>numerOfReads
>TagList
>otherStuff
>
>
>Then I look up a relationship entity to find out everyone who is a
>FRIEND of mine.
>FRIENDS:
>userKey
>friendKey
>
>Then I look up a relationship entity to find out which BOOKS that
>USERS have read
>BOOKSREAD:
>userKey
>bookKey
>TagList
>numberOfReads
>
>So first I pull into memory the list of my particular friendKeys from
>the FRIENDS table.
>
>Then I churn through the table of BOOKSREAD, using the duplicated
>properties TagList and numberOfReads to filter on tags and sort
>according to number of reads.
>
>I then filter this list of BOOKS *in memory* by the friendKeys I
>have. Ill need some sort of algorithm to keep pulling chunks of BOOKS
>from the database until I have a nice page of books my friends have
>read (20 books displayed per page) to pass to the GUI.
>
>????
>
>--
>You received this message because you are subscribed to the Google Groups
>"Google App Engine" group.
>To post to this group, send email to [email protected].
>To unsubscribe from this group, send email to
>[email protected].
>For more options, visit this group at
>http://groups.google.com/group/google-appengine?hl=en.
>
>
--
--
Jeff
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
