Also, I think we can add RoR-like migrations on top of that with automatic schema versioning and all the stuff.
Best regards, Andrey Martyanov On Fri, Mar 2, 2012 at 6:22 PM, Andrey V. Martyanov <reald...@gmail.com>wrote: > I'm sorry for the repost, but my gmail client by default sends an email to > a specific subscriber, not to riak-users. > > > Hi Shuhao, > > I don't have enough time to write my own implementation and to see how it > works in practice. But I share my thoughts about that. > It's possible to save the current API, it's common now and used in > SQLAlchemy and Django ORM. But underlying implementation I see a bit > different. The simple workflow: > 1. Define the models with specific fields and properties; > 2. Generate the JSON Schema [1] based on these fields; > 3. Create an instances of our models; > 4. Validate against the schema; > 5. Generate the optimal sequence of actions we need to do to store the > data, based on the meta information of our models; > 6. Store the data, resolve the conflicts if occur. > > Also it should be possible to deserialize the JSON objects into python > objects. > > [1] - http://json-schema.org/ > > Best regards, > Andrey Martyanov > > On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <reald...@gmail.com>wrote: > >> Hi Shuhao, >> >> I don't have enough time to write my own implementation and to see how it >> works in practice. But I share my thoughts about that. >> It's possible to save the current API, it's common now and used in >> SQLAlchemy and Django ORM. But underlying implementation I see a bit >> different. The simple workflow: >> 1. Define the models with specific fields and properties; >> 2. Generate the JSON Schema [1] based on these fields; >> 3. Create an instances of our models; >> 4. Validate against the schema; >> 5. Generate the optimal sequence of actions we need to do to store the >> data, based on the meta information of our models; >> 6. Store the data, resolve the conflicts if occur. >> >> Also it should be possible to deserialize the JSON objects into python >> objects. >> >> [1] - http://json-schema.org/ >> >> Best regards, >> Andrey Martyanov >> >>> >>> >>> On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <ad...@thekks.net> wrote: >>> >>>> Yeah. You're looking at something like dictshield. As riakkit is more a >>>> RAD tool in its current state. You bring in an excellent point, >>>> nonetheless, which gave me some intuition and ideas about the next step. >>>> >>>> Cheers, >>>> Shuhao >>>> Sent from my phone. >>>> On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <reald...@gmail.com> >>>> wrote: >>>> >>>>> Riakkit in its current implementation has a lot of overhead. But what >>>>> I get for this price? Almost nothing. Only field validation make sense. >>>>> Simple key-value mapping is about 50-70 lines of code, not a big deal, >>>>> even >>>>> for small project. Riak used mostly in web-based projects. Almost all >>>>> modern projects use JSON. The simple workflow is get JSON from the client, >>>>> validate, store in the database, query the database, send JSON to the >>>>> client. If our mapper doesn't understand the JSON and cannot operate on it >>>>> then it's not useful. We can use jsonpickle with a lot more profit. Just >>>>> jsonpickle.encode the object and store it in Riak. I'm sorry, this is my >>>>> position! >>>>> >>>>> Best regards, >>>>> Andrey Martyanov >>>>> >>>>> On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov < >>>>> reald...@gmail.com> wrote: >>>>> >>>>>> I'm just saying that to efficiently store and get the data user >>>>>> should have ability to customize the relationship. Riak has several >>>>>> options >>>>>> to get related data: secondary indexes, map-reduce etc. I think that the >>>>>> most important task of mapper is to validate the data and to track these >>>>>> dependencies. When you have enough meta information about objects storing >>>>>> these objects is not a big problem. >>>>>> >>>>>> >>>>>> On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <ad...@thekks.net> wrote: >>>>>> >>>>>>> The thing about riak-python-client is that it's already pretty high >>>>>>> level, but not high level enough to be that simple. >>>>>>> >>>>>>> From what I understand, tracking relationship between models will >>>>>>> still involve a certain level of magic. Right now the magic is that when >>>>>>> you save any document, all the related document will be saved into the >>>>>>> database as well. You're saying that just tracking those relationship >>>>>>> change is good enough and saving will be up to the user? >>>>>>> >>>>>>> Shuhao >>>>>>> >>>>>>> >>>>>>> On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov < >>>>>>> reald...@gmail.com> wrote: >>>>>>> >>>>>>>> I'm not sure about forking the riak-python-client. The the whole >>>>>>>> idea of separation the low-level client library from the high-level >>>>>>>> mapper >>>>>>>> and the like is quite good. Some sort of SQLAlchemy does. >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Andrey Martyanov >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <ad...@thekks.net> wrote: >>>>>>>> >>>>>>>>> Another thing I was considering is forking riak-python-client. >>>>>>>>> Some of the stuff in there could use some fixing as well, and maybe >>>>>>>>> just >>>>>>>>> take out the core transports and such and plug that right into >>>>>>>>> riakkit and >>>>>>>>> skip the notion of RiakObject all together as those API is also >>>>>>>>> somewhat >>>>>>>>> weird. (such as add_link and remove_link) >>>>>>>>> >>>>>>>>> Shuhao >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <ad...@thekks.net>wrote: >>>>>>>>> >>>>>>>>>> Hi! >>>>>>>>>> >>>>>>>>>> First off, the code is *LGPL* licensed, not GPL. So you're free >>>>>>>>>> to put this into any code you want, it's just that when you want to >>>>>>>>>> modify >>>>>>>>>> riakkit, you have to release your modified version. >>>>>>>>>> >>>>>>>>>> I just went over your idea and it would be a valid solution to >>>>>>>>>> some of the problems I'm having right now with my own projects with >>>>>>>>>> riakkit. When I initially started writing riakkit, I needed >>>>>>>>>> something to do >>>>>>>>>> RAD with, which would make sense to have an all-in-one, very simple >>>>>>>>>> to use >>>>>>>>>> framework, rather than working with serialization and what not. >>>>>>>>>> However, >>>>>>>>>> now I'm beginning to see problems in this approach when it comes to >>>>>>>>>> scalability as I'm slowing moving out of the RAD phase. I haven't >>>>>>>>>> done >>>>>>>>>> anything as I have not encountered any major issues yet (though I >>>>>>>>>> probably >>>>>>>>>> will). >>>>>>>>>> >>>>>>>>>> So here's what I'm thinking for the next branch ( I still have to >>>>>>>>>> maintain the master branch due to some code i've written and need ): >>>>>>>>>> I'm >>>>>>>>>> going to break riakkit into 2 pieces. The first piece is the >>>>>>>>>> underlying >>>>>>>>>> piece, which will do what you said, serialize/deserialize objects, >>>>>>>>>> validation, track relationships and what not, however, it doesn't >>>>>>>>>> load it >>>>>>>>>> those references up nor does it try to accomplish everything all in >>>>>>>>>> one >>>>>>>>>> shot, and a RAD layer that's compatible with the current API that >>>>>>>>>> will >>>>>>>>>> allow RAD to work with it, but still reserve the flexibility of >>>>>>>>>> having that >>>>>>>>>> underlying API that's relatively easy (as oppose to dead simple) to >>>>>>>>>> work >>>>>>>>>> with and not as messy as what's going on right now. >>>>>>>>>> >>>>>>>>>> I do agree that the code is messy.. Most of the time it's me >>>>>>>>>> fixing bugs I didn't anticipate as I go along and hacking everything >>>>>>>>>> into >>>>>>>>>> place.. >>>>>>>>>> >>>>>>>>>> One thing I'm not too clear about is what you mean by "Track >>>>>>>>>> dependencies between models". I'm not sure how you think I should go >>>>>>>>>> about >>>>>>>>>> approaching this (which right now is a pretty big disaster.. though >>>>>>>>>> relatively stable) >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> >>>>>>>>>> Shuhao >>>>>>>>>> >>>>>>>>>> P.S. I don't want to go into the whole PEP8 discussion and 4 >>>>>>>>>> spaces vs whatever.. I just like the 2 spaces and the style guide >>>>>>>>>> that >>>>>>>>>> Google uses. Is that really a big deal if you're using it as a >>>>>>>>>> library? >>>>>>>>>> >>>>>>>>>> On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov < >>>>>>>>>> reald...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi! >>>>>>>>>>> >>>>>>>>>>> Two things force me to avoid using Riakkit: the code is GPL >>>>>>>>>>> licensed and non pep8 compliant. So, I cannot fork it and make >>>>>>>>>>> changes. The >>>>>>>>>>> code needs clean up, for example, json module is imported in >>>>>>>>>>> document.py, >>>>>>>>>>> but not used. Riakkit has no conflict resolution logic, it's >>>>>>>>>>> basically >>>>>>>>>>> delegate the work to underlying python client. If I'd like to >>>>>>>>>>> implement my >>>>>>>>>>> save method I need to copy and rewrite more than 150 lines of code >>>>>>>>>>> mess. >>>>>>>>>>> What I think the mapper should do: >>>>>>>>>>> 1. Serialize the objects to valid json; >>>>>>>>>>> 2. Deserialize the data back to class instances; >>>>>>>>>>> 3. Validate the fields data; >>>>>>>>>>> 4. Track dependencies between models; >>>>>>>>>>> >>>>>>>>>>> Simple example using Riakkit-like syntax; >>>>>>>>>>> >>>>>>>>>>> class Post(Document): >>>>>>>>>>> >>>>>>>>>>> bucket_name = 'posts' >>>>>>>>>>> >>>>>>>>>>> title = types.StringProperty(required=True) >>>>>>>>>>> content = types.StringProperty(required=True) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> class Comment(Document): >>>>>>>>>>> >>>>>>>>>>> bucket_name = 'comments' >>>>>>>>>>> >>>>>>>>>>> text = types.StringProperty(required=True) >>>>>>>>>>> post = types.IndexReferenceProperty(reference_class=Post, >>>>>>>>>>> collection_name='comments') >>>>>>>>>>> >>>>>>>>>>> >>> post = Post('title', 'content') >>>>>>>>>>> >>> comment1 = ('comment1', post=post) >>>>>>>>>>> >>> comment2 = ('comment2', post=post) >>>>>>>>>>> >>> post.is_valid() >>>>>>>>>>> True >>>>>>>>>>> >> post.serialize() >>>>>>>>>>> { >>>>>>>>>>> "key": "post_key", >>>>>>>>>>> "title": "title", >>>>>>>>>>> "content": "content", >>>>>>>>>>> "comments": [ >>>>>>>>>>> { >>>>>>>>>>> "key": "comment1_key", >>>>>>>>>>> "text": "comment1" >>>>>>>>>>> }, >>>>>>>>>>> { >>>>>>>>>>> "key": "comment2_key", >>>>>>>>>>> "text": "comment2" >>>>>>>>>>> }, >>>>>>>>>>> ] >>>>>>>>>>> } >>>>>>>>>>> >>> post.store() >>>>>>>>>>> True >>>>>>>>>>> >>> post = Post.get_with_key('post_key') >>>>>>>>>>> >>> post.comments >>>>>>>>>>> ['Post comment1', 'Post comment2'] >>>>>>>>>>> >>> post_json_data = get_json_data_from_request() >>>>>>>>>>> >>> post = Post.deserialize(post_json_data) >>>>>>>>>>> >>> post.title >>>>>>>>>>> 'title' >>>>>>>>>>> >>> post.comments >>>>>>>>>>> ['Post comment1', 'Post comment2'] >>>>>>>>>>> >>>>>>>>>>> Something like that. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Best regards, >>>>>>>>>>> Andrey Martyanov >>>>>>>>>>> >>>>>>>>>>> On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <ca...@basho.com>wrote: >>>>>>>>>>> >>>>>>>>>>>> Keep up the great work Shuhao! >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mar 1, 2012, at 7:36 PM, Shuhao Wu <ad...@thekks.net> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hey guys, >>>>>>>>>>>> >>>>>>>>>>>> I started writing riakkit approximately 3 months ago and >>>>>>>>>>>> announced it here. In case you missed that, Riakkit is basically >>>>>>>>>>>> an object >>>>>>>>>>>> mapper for python and riak, which seemed to be lacking in at the >>>>>>>>>>>> time. Now >>>>>>>>>>>> that 3 months has passed and I have developed this project into >>>>>>>>>>>> much more >>>>>>>>>>>> than what I had envisioned to begin with. It's currently also >>>>>>>>>>>> being used >>>>>>>>>>>> and tested in a project of mine (web application) as well and >>>>>>>>>>>> functioning >>>>>>>>>>>> quite nicely. I would say that currently this is the only object >>>>>>>>>>>> mapper >>>>>>>>>>>> that can compete with their counterparts in other languages for >>>>>>>>>>>> Riak in >>>>>>>>>>>> Python, although a lot of work still needs to be done before it >>>>>>>>>>>> could fully >>>>>>>>>>>> compete. >>>>>>>>>>>> >>>>>>>>>>>> So today I felt that riakkit is "stable" and "clean" enough to >>>>>>>>>>>> deserve the first beta release. So I whipped up a website and it's >>>>>>>>>>>> released >>>>>>>>>>>> to pypi. Here's (again), a demo (the full tutorial can be found on >>>>>>>>>>>> the >>>>>>>>>>>> links below) >>>>>>>>>>>> >>>>>>>>>>>> class User(Document): >>>>>>>>>>>> client = riak.RiakClient() >>>>>>>>>>>> bucket_name = "users" >>>>>>>>>>>> >>>>>>>>>>>> name = StringProperty(required=True) >>>>>>>>>>>> >>>>>>>>>>>> >>> user = User(name="foo") >>>>>>>>>>>> >>> user.save() >>>>>>>>>>>> >>> print user.name >>>>>>>>>>>> foo >>>>>>>>>>>> >>>>>>>>>>>> If anyone is interested, you can find the website here: >>>>>>>>>>>> http://ultimatebuster.github.com/riakkit and the code at >>>>>>>>>>>> http://github.com/ultimatebuster/riakkit >>>>>>>>>>>> >>>>>>>>>>>> Cheers! >>>>>>>>>>>> >>>>>>>>>>>> Shuhao >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> riak-users mailing list >>>>>>>>>>>> riak-users@lists.basho.com >>>>>>>>>>>> >>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> riak-users mailing list >>>>>>>>>>>> riak-users@lists.basho.com >>>>>>>>>>>> >>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> riak-users mailing list >>>>>>>>>>> riak-users@lists.basho.com >>>>>>>>>>> >>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> riak-users mailing list >>>>> riak-users@lists.basho.com >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>> >>>>> >>> >> >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
