Also, what do you mean by conflict resolution? Synchronization or conflicts between references such as deleting references to a document when that document is deleted?
Shuhao On Fri, Mar 2, 2012 at 9:56 AM, Shuhao Wu <ad...@thekks.net> wrote: > Actually it's okay for no pull request for now.. I'm in the process of > rethinking the whole framework as well as I'm having some of the issues > that Andrey is describing as well. > > Shuhao > > > > On Fri, Mar 2, 2012 at 9:51 AM, Jared Morrow <ja...@basho.com> wrote: > >> Andrey, >> >> First off, you are indeed sending two emails to the Riak Users list each >> time you email, not just one, so I think gmail is doing the right thing the >> first time. >> >> Second, I'm sure Shuhao would love some pull requests to help with >> Riakkit since you seem to have a lot of ideas to make it better. >> >> -Jared >> >> >> On Mar 2, 2012, at 6:39 AM, Andrey V. Martyanov wrote: >> >> Also, I think we can add RoR-like migrations on top of that with >> automatic schema versioning and all the stuff. >> >> Best regards, >> Andrey Martyanov >> >> On Fri, Mar 2, 2012 at 6:22 PM, Andrey V. Martyanov >> <reald...@gmail.com>wrote: >> >>> I'm sorry for the repost, but my gmail client by default sends an email >>> to a specific subscriber, not to riak-users. >>> >>> >>> Hi Shuhao, >>> >>> I don't have enough time to write my own implementation and to see how >>> it works in practice. But I share my thoughts about that. >>> It's possible to save the current API, it's common now and used in >>> SQLAlchemy and Django ORM. But underlying implementation I see a bit >>> different. The simple workflow: >>> 1. Define the models with specific fields and properties; >>> 2. Generate the JSON Schema [1] based on these fields; >>> 3. Create an instances of our models; >>> 4. Validate against the schema; >>> 5. Generate the optimal sequence of actions we need to do to store the >>> data, based on the meta information of our models; >>> 6. Store the data, resolve the conflicts if occur. >>> >>> Also it should be possible to deserialize the JSON objects into python >>> objects. >>> >>> [1] - http://json-schema.org/ >>> >>> Best regards, >>> Andrey Martyanov >>> >>> On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov >>> <reald...@gmail.com>wrote: >>> >>>> Hi Shuhao, >>>> >>>> I don't have enough time to write my own implementation and to see how >>>> it works in practice. But I share my thoughts about that. >>>> It's possible to save the current API, it's common now and used in >>>> SQLAlchemy and Django ORM. But underlying implementation I see a bit >>>> different. The simple workflow: >>>> 1. Define the models with specific fields and properties; >>>> 2. Generate the JSON Schema [1] based on these fields; >>>> 3. Create an instances of our models; >>>> 4. Validate against the schema; >>>> 5. Generate the optimal sequence of actions we need to do to store the >>>> data, based on the meta information of our models; >>>> 6. Store the data, resolve the conflicts if occur. >>>> >>>> Also it should be possible to deserialize the JSON objects into python >>>> objects. >>>> >>>> [1] - http://json-schema.org/ >>>> >>>> Best regards, >>>> Andrey Martyanov >>>> >>>>> >>>>> >>>>> On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <ad...@thekks.net> wrote: >>>>> >>>>>> Yeah. You're looking at something like dictshield. As riakkit is more >>>>>> a RAD tool in its current state. You bring in an excellent point, >>>>>> nonetheless, which gave me some intuition and ideas about the next step. >>>>>> >>>>>> Cheers, >>>>>> Shuhao >>>>>> Sent from my phone. >>>>>> On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <reald...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Riakkit in its current implementation has a lot of overhead. But >>>>>>> what I get for this price? Almost nothing. Only field validation make >>>>>>> sense. Simple key-value mapping is about 50-70 lines of code, not a big >>>>>>> deal, even for small project. Riak used mostly in web-based projects. >>>>>>> Almost all modern projects use JSON. The simple workflow is get JSON >>>>>>> from >>>>>>> the client, validate, store in the database, query the database, send >>>>>>> JSON >>>>>>> to the client. If our mapper doesn't understand the JSON and cannot >>>>>>> operate >>>>>>> on it then it's not useful. We can use jsonpickle with a lot more >>>>>>> profit. >>>>>>> Just jsonpickle.encode the object and store it in Riak. I'm sorry, this >>>>>>> is >>>>>>> my position! >>>>>>> >>>>>>> Best regards, >>>>>>> Andrey Martyanov >>>>>>> >>>>>>> On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov < >>>>>>> reald...@gmail.com> wrote: >>>>>>> >>>>>>>> I'm just saying that to efficiently store and get the data user >>>>>>>> should have ability to customize the relationship. Riak has several >>>>>>>> options >>>>>>>> to get related data: secondary indexes, map-reduce etc. I think that >>>>>>>> the >>>>>>>> most important task of mapper is to validate the data and to track >>>>>>>> these >>>>>>>> dependencies. When you have enough meta information about objects >>>>>>>> storing >>>>>>>> these objects is not a big problem. >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <ad...@thekks.net> wrote: >>>>>>>> >>>>>>>>> The thing about riak-python-client is that it's already pretty >>>>>>>>> high level, but not high level enough to be that simple. >>>>>>>>> >>>>>>>>> From what I understand, tracking relationship between models will >>>>>>>>> still involve a certain level of magic. Right now the magic is that >>>>>>>>> when >>>>>>>>> you save any document, all the related document will be saved into the >>>>>>>>> database as well. You're saying that just tracking those relationship >>>>>>>>> change is good enough and saving will be up to the user? >>>>>>>>> >>>>>>>>> Shuhao >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov < >>>>>>>>> reald...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> I'm not sure about forking the riak-python-client. The the whole >>>>>>>>>> idea of separation the low-level client library from the high-level >>>>>>>>>> mapper >>>>>>>>>> and the like is quite good. Some sort of SQLAlchemy does. >>>>>>>>>> >>>>>>>>>> Best regards, >>>>>>>>>> Andrey Martyanov >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <ad...@thekks.net>wrote: >>>>>>>>>> >>>>>>>>>>> Another thing I was considering is forking riak-python-client. >>>>>>>>>>> Some of the stuff in there could use some fixing as well, and maybe >>>>>>>>>>> just >>>>>>>>>>> take out the core transports and such and plug that right into >>>>>>>>>>> riakkit and >>>>>>>>>>> skip the notion of RiakObject all together as those API is also >>>>>>>>>>> somewhat >>>>>>>>>>> weird. (such as add_link and remove_link) >>>>>>>>>>> >>>>>>>>>>> Shuhao >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <ad...@thekks.net>wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi! >>>>>>>>>>>> >>>>>>>>>>>> First off, the code is *LGPL* licensed, not GPL. So you're >>>>>>>>>>>> free to put this into any code you want, it's just that when you >>>>>>>>>>>> want to >>>>>>>>>>>> modify riakkit, you have to release your modified version. >>>>>>>>>>>> >>>>>>>>>>>> I just went over your idea and it would be a valid solution to >>>>>>>>>>>> some of the problems I'm having right now with my own projects with >>>>>>>>>>>> riakkit. When I initially started writing riakkit, I needed >>>>>>>>>>>> something to do >>>>>>>>>>>> RAD with, which would make sense to have an all-in-one, very >>>>>>>>>>>> simple to use >>>>>>>>>>>> framework, rather than working with serialization and what not. >>>>>>>>>>>> However, >>>>>>>>>>>> now I'm beginning to see problems in this approach when it comes to >>>>>>>>>>>> scalability as I'm slowing moving out of the RAD phase. I haven't >>>>>>>>>>>> done >>>>>>>>>>>> anything as I have not encountered any major issues yet (though I >>>>>>>>>>>> probably >>>>>>>>>>>> will). >>>>>>>>>>>> >>>>>>>>>>>> So here's what I'm thinking for the next branch ( I still have >>>>>>>>>>>> to maintain the master branch due to some code i've written and >>>>>>>>>>>> need ): I'm >>>>>>>>>>>> going to break riakkit into 2 pieces. The first piece is the >>>>>>>>>>>> underlying >>>>>>>>>>>> piece, which will do what you said, serialize/deserialize objects, >>>>>>>>>>>> validation, track relationships and what not, however, it doesn't >>>>>>>>>>>> load it >>>>>>>>>>>> those references up nor does it try to accomplish everything all >>>>>>>>>>>> in one >>>>>>>>>>>> shot, and a RAD layer that's compatible with the current API that >>>>>>>>>>>> will >>>>>>>>>>>> allow RAD to work with it, but still reserve the flexibility of >>>>>>>>>>>> having that >>>>>>>>>>>> underlying API that's relatively easy (as oppose to dead simple) >>>>>>>>>>>> to work >>>>>>>>>>>> with and not as messy as what's going on right now. >>>>>>>>>>>> >>>>>>>>>>>> I do agree that the code is messy.. Most of the time it's me >>>>>>>>>>>> fixing bugs I didn't anticipate as I go along and hacking >>>>>>>>>>>> everything into >>>>>>>>>>>> place.. >>>>>>>>>>>> >>>>>>>>>>>> One thing I'm not too clear about is what you mean by "Track >>>>>>>>>>>> dependencies between models". I'm not sure how you think I should >>>>>>>>>>>> go about >>>>>>>>>>>> approaching this (which right now is a pretty big disaster.. though >>>>>>>>>>>> relatively stable) >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> >>>>>>>>>>>> Shuhao >>>>>>>>>>>> >>>>>>>>>>>> P.S. I don't want to go into the whole PEP8 discussion and 4 >>>>>>>>>>>> spaces vs whatever.. I just like the 2 spaces and the style guide >>>>>>>>>>>> that >>>>>>>>>>>> Google uses. Is that really a big deal if you're using it as a >>>>>>>>>>>> library? >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov < >>>>>>>>>>>> reald...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi! >>>>>>>>>>>>> >>>>>>>>>>>>> Two things force me to avoid using Riakkit: the code is GPL >>>>>>>>>>>>> licensed and non pep8 compliant. So, I cannot fork it and make >>>>>>>>>>>>> changes. The >>>>>>>>>>>>> code needs clean up, for example, json module is imported in >>>>>>>>>>>>> document.py, >>>>>>>>>>>>> but not used. Riakkit has no conflict resolution logic, it's >>>>>>>>>>>>> basically >>>>>>>>>>>>> delegate the work to underlying python client. If I'd like to >>>>>>>>>>>>> implement my >>>>>>>>>>>>> save method I need to copy and rewrite more than 150 lines of >>>>>>>>>>>>> code mess. >>>>>>>>>>>>> What I think the mapper should do: >>>>>>>>>>>>> 1. Serialize the objects to valid json; >>>>>>>>>>>>> 2. Deserialize the data back to class instances; >>>>>>>>>>>>> 3. Validate the fields data; >>>>>>>>>>>>> 4. Track dependencies between models; >>>>>>>>>>>>> >>>>>>>>>>>>> Simple example using Riakkit-like syntax; >>>>>>>>>>>>> >>>>>>>>>>>>> class Post(Document): >>>>>>>>>>>>> >>>>>>>>>>>>> bucket_name = 'posts' >>>>>>>>>>>>> >>>>>>>>>>>>> title = types.StringProperty(required=True) >>>>>>>>>>>>> content = types.StringProperty(required=True) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> class Comment(Document): >>>>>>>>>>>>> >>>>>>>>>>>>> bucket_name = 'comments' >>>>>>>>>>>>> >>>>>>>>>>>>> text = types.StringProperty(required=True) >>>>>>>>>>>>> post = types.IndexReferenceProperty(reference_class=Post, >>>>>>>>>>>>> collection_name='comments') >>>>>>>>>>>>> >>>>>>>>>>>>> >>> post = Post('title', 'content') >>>>>>>>>>>>> >>> comment1 = ('comment1', post=post) >>>>>>>>>>>>> >>> comment2 = ('comment2', post=post) >>>>>>>>>>>>> >>> post.is_valid() >>>>>>>>>>>>> True >>>>>>>>>>>>> >> post.serialize() >>>>>>>>>>>>> { >>>>>>>>>>>>> "key": "post_key", >>>>>>>>>>>>> "title": "title", >>>>>>>>>>>>> "content": "content", >>>>>>>>>>>>> "comments": [ >>>>>>>>>>>>> { >>>>>>>>>>>>> "key": "comment1_key", >>>>>>>>>>>>> "text": "comment1" >>>>>>>>>>>>> }, >>>>>>>>>>>>> { >>>>>>>>>>>>> "key": "comment2_key", >>>>>>>>>>>>> "text": "comment2" >>>>>>>>>>>>> }, >>>>>>>>>>>>> ] >>>>>>>>>>>>> } >>>>>>>>>>>>> >>> post.store() >>>>>>>>>>>>> True >>>>>>>>>>>>> >>> post = Post.get_with_key('post_key') >>>>>>>>>>>>> >>> post.comments >>>>>>>>>>>>> ['Post comment1', 'Post comment2'] >>>>>>>>>>>>> >>> post_json_data = get_json_data_from_request() >>>>>>>>>>>>> >>> post = Post.deserialize(post_json_data) >>>>>>>>>>>>> >>> post.title >>>>>>>>>>>>> 'title' >>>>>>>>>>>>> >>> post.comments >>>>>>>>>>>>> ['Post comment1', 'Post comment2'] >>>>>>>>>>>>> >>>>>>>>>>>>> Something like that. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Best regards, >>>>>>>>>>>>> Andrey Martyanov >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <ca...@basho.com>wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Keep up the great work Shuhao! >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mar 1, 2012, at 7:36 PM, Shuhao Wu <ad...@thekks.net> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hey guys, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I started writing riakkit approximately 3 months ago and >>>>>>>>>>>>>> announced it here. In case you missed that, Riakkit is basically >>>>>>>>>>>>>> an object >>>>>>>>>>>>>> mapper for python and riak, which seemed to be lacking in at the >>>>>>>>>>>>>> time. Now >>>>>>>>>>>>>> that 3 months has passed and I have developed this project into >>>>>>>>>>>>>> much more >>>>>>>>>>>>>> than what I had envisioned to begin with. It's currently also >>>>>>>>>>>>>> being used >>>>>>>>>>>>>> and tested in a project of mine (web application) as well and >>>>>>>>>>>>>> functioning >>>>>>>>>>>>>> quite nicely. I would say that currently this is the only object >>>>>>>>>>>>>> mapper >>>>>>>>>>>>>> that can compete with their counterparts in other languages for >>>>>>>>>>>>>> Riak in >>>>>>>>>>>>>> Python, although a lot of work still needs to be done before it >>>>>>>>>>>>>> could fully >>>>>>>>>>>>>> compete. >>>>>>>>>>>>>> >>>>>>>>>>>>>> So today I felt that riakkit is "stable" and "clean" enough >>>>>>>>>>>>>> to deserve the first beta release. So I whipped up a website and >>>>>>>>>>>>>> it's >>>>>>>>>>>>>> released to pypi. Here's (again), a demo (the full tutorial can >>>>>>>>>>>>>> be found on >>>>>>>>>>>>>> the links below) >>>>>>>>>>>>>> >>>>>>>>>>>>>> class User(Document): >>>>>>>>>>>>>> client = riak.RiakClient() >>>>>>>>>>>>>> bucket_name = "users" >>>>>>>>>>>>>> >>>>>>>>>>>>>> name = StringProperty(required=True) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>> user = User(name="foo") >>>>>>>>>>>>>> >>> user.save() >>>>>>>>>>>>>> >>> print user.name >>>>>>>>>>>>>> foo >>>>>>>>>>>>>> >>>>>>>>>>>>>> If anyone is interested, you can find the website here: >>>>>>>>>>>>>> http://ultimatebuster.github.com/riakkit and the code at >>>>>>>>>>>>>> http://github.com/ultimatebuster/riakkit >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Shuhao >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> riak-users mailing list >>>>>>>>>>>>>> riak-users@lists.basho.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> riak-users mailing list >>>>>>>>>>>>>> riak-users@lists.basho.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> riak-users mailing list >>>>>>>>>>>>> riak-users@lists.basho.com >>>>>>>>>>>>> >>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> riak-users mailing list >>>>>>> riak-users@lists.basho.com >>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>> >>>>>>> >>>>> >>>> >>> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
