RE work in progress code The main functions to optimize are get_new_fields, get_new_field, they are found here: https://github.com/PirosB3/django/blob/a92c37f0cad6bdd7a3b24ef235c81a7dab6bee72/django/db/models/options.py
On Sunday, June 22, 2014 11:49:05 AM UTC+2, Daniel Pyrathon wrote: > > Sure! > > > https://github.com/PirosB3/django/compare/soc2014_meta_refactor_performance?expand=1 > > This contains the following: > - Recently merged unit test suite for model/options > - New API implementation: get_new_fields, get_new_field. > - Implementation of recently merged unit test suite with new API > - Small optimizations (BIG work in progress) > > If you have any suggestions please let me know! > > Dan > > On Sunday, June 22, 2014 2:14:56 AM UTC+2, Curtis Maloney wrote: >> >> Is there somewhere I can look at your work in progress code? >> >> >> On 21 June 2014 19:57, Daniel Pyrathon <[email protected]> wrote: >> >>> Hi All, >>> >>> This week I have done the following >>> >>> *- Openend PR and merged Options (_meta) unittests* >>> https://github.com/django/django/pull/2823 >>> This is a working test suite of model/options.py file. Is is currently >>> testing the current Options implementation. >>> >>> *- Re-implemented test-suite in the new API* >>> My new proposed API has been implemented on top of the Options >>> unittests. This means new and old API give the same results! and therefore >>> the new API is a working re-implementation. >>> >>> *- Performance optimizations* >>> Currently, the biggest issue I have had (in the new API) regards >>> performance. I am doing small optimisations and benchmarking with cProfile. >>> If anyone has some good hints on how to benchmark single functions in >>> Django please let me know! >>> >>> Regards, >>> Dan >>> >>> >>> On Friday, June 13, 2014 10:11:41 PM UTC+2, Aymeric Augustin wrote: >>> >>>> I like this simplification a lot. I hope you can make it work. >>>> >>>> Just check that you don’t "overload" fields, by storing in field >>>> objects information that doesn’t belong there. >>>> >>>> -- >>>> Aymeric. >>>> >>>> >>>> >>>> On 13 juin 2014, at 19:53, Daniel Pyrathon <[email protected]> wrote: >>>> >>>> Hi All, >>>> >>>> This week's work was a follow-up of last week's work, with some new >>>> discoveries with regards to the new API: >>>> >>>> *1) Improved the current _meta implementation of unittests* >>>> I refactored the current meta unittest branch into a more compact >>>> version. I also added test cases for Generic Foreign Keys, RelatedField >>>> and >>>> more improvements on virtual fields. This section will actually be our >>>> first merge to master: a unit test suite for the current implementation. >>>> >>>> This branch is found here: https://github.com/ >>>> PirosB3/django/tree/soc2014_meta_unittests >>>> >>>> *2) Refactored the new _meta API spec* >>>> By implementing the new API, I found new redundancies in the current >>>> implementation that we can avoid in the new API spec: >>>> >>>> *1) Only return field instances* >>>> the current implementation has a common return pattern: (field_object, >>>> model, direct, m2m). After a few tests I realized that model, direct, m2m >>>> and field_name are all redundant and can be derived from only >>>> field_object. >>>> Since there are only 3-4 places that actually use direct and m2m it makes >>>> sense to remove this function from the new API spec. >>>> Here I show how all the information can be derived form field_instance ( >>>> https://gist.github.com/PirosB3/6cb4badbb1b8c2e41a96/revisions), I ran >>>> the entire test suite and things look good. The only issue I can see now >>>> is >>>> from a performance point of view. >>>> >>>> *2) Provide only 2 API endpoints* >>>> >>>> - get_fields(types, opts) -> (field_instance, field_instance, ...) >>>> - get_field(field_name) -> field_instance >>>> >>>> The rest is all (I might be very wrong! don't trust me) redundant. By >>>> continuing with this iterative approach, I will soon find out if I am >>>> correct or not ;) >>>> >>>> >>>> This branch is found here: https://github.com/ >>>> PirosB3/django/tree/soc2014_meta_refactor >>>> >>>> For any questions, as usual, let me know >>>> Dan >>>> >>>> >>>> On Friday, June 6, 2014 7:03:36 PM UTC+2, Daniel Pyrathon wrote: >>>>> >>>>> Hi All, >>>>> >>>>> Based on the work done last week, this week I have worked on the >>>>> following: >>>>> >>>>> *1) Covered the current _meta implementation of unittests* >>>>> The current Options is not covered by unit tests at all, I have >>>>> created the model_options test module containing one or more unit tests >>>>> for >>>>> each endpoint and usage. Each endpoint is tested with many different >>>>> models >>>>> and fields (data, m2m, related objects, related m2m, virtual, ..). Each >>>>> unit test asserts equality in field naming and field type. Endpoints that >>>>> return the model are also tested for equality. >>>>> >>>>> This branch is found here: https://github.com/ >>>>> PirosB3/django/tree/soc2014_meta_unittests >>>>> >>>>> *2) Pulled in tests from soc2014_meta_unittests and tested the new >>>>> implementation* >>>>> The previous branch that I developed on contains the new API >>>>> implementation, I have pulled in the tests from soc2014_meta_unittests >>>>> and >>>>> I have switched the old API calls to the new API calls (with a few >>>>> adjustments). I have successfully made all tests pass even though I have >>>>> found some design issues that need to be addressed in my next update call. >>>>> >>>>> This branch is found here: https://github.com/ >>>>> PirosB3/django/tree/soc2014_meta_refactor >>>>> >>>>> *3) Created a new branch that maps the old implementation with the new* >>>>> Today I started putting my new API in "production". This is obviously >>>>> nowhere near a finalised version but it helps me spot some edge cases >>>>> that >>>>> are not in the unit-tests. Each issue found has been converted into a >>>>> standalone unit-test and has been proved to fail. >>>>> Unfortunately, this made me realise of other design issues related to >>>>> the new return type of get_fields -> (field_name, field), A decision will >>>>> be taken on monday. >>>>> >>>>> This branch is found here: https://github.com/ >>>>> PirosB3/django/tree/soc2014_meta_refactor_implementation as is for >>>>> personal use only >>>>> >>>>> For any questions, let me know! >>>>> Dan >>>>> >>>>> >>>>> On Sunday, June 1, 2014 6:10:53 PM UTC+2, Daniel Pyrathon wrote: >>>>>> >>>>>> Hi All, >>>>>> >>>>>> An update on my side, some interesting work has happened this week: >>>>>> me and Russell have decided to start on the implementation early in >>>>>> order >>>>>> to understand better the internals of Options. Currently I am working on >>>>>> the following: >>>>>> >>>>>> *Providing one single function: get_fields(types, opts, **kwargs)* >>>>>> Previously we had identified a number of functions that contained >>>>>> similar data but had a different return type. We are going to provide >>>>>> one >>>>>> function that takes a set of field types + options and returns the same >>>>>> output everywhere: ((field_name, field), ...). This has the benefit of >>>>>> simplicity and is more maintainable than the previous approach. >>>>>> >>>>>> TYPES = DATA, M2M, FK, RELATED_OBJECT, RELATED_M2M >>>>>> OPTS = NONE, LOCAL_ONLY, CONCRETE, INCLUDE_HIDDEN, INCLUDE_PROXY, >>>>>> VIRTUAL >>>>>> >>>>>> *Providing two functions for retrieving details of a specific field* >>>>>> As specified in my previous document, in many parts of the code we >>>>>> just want to retrieve a field object by name, other times we have a >>>>>> field >>>>>> but need other metadata such as: owner (model_class), direct (bool), m2m >>>>>> (bool). We will provide two functions: >>>>>> >>>>>> get_field(field_name) -> field_instance >>>>>> >>>>>> get_field_details(field_instance) -> direct, m2m, (still to be >>>>>> defined) >>>>>> >>>>>> While we still have not entirely defined what *get_field_details *will >>>>>> return, this will be done soon. >>>>>> >>>>>> >>>>>> *Building a test suite for the existing API* >>>>>> The new API development will be driven by a test suite that will >>>>>> compare the current (legacy) API with the new implementation. While >>>>>> return >>>>>> types will be different, we are asserting that all the correct fields >>>>>> and >>>>>> metadata are returned. Building a test suite means we can start >>>>>> implementing before a final API spec is finalised. It also means we can >>>>>> iterate faster and, from my perspective, I also understand a lot more of >>>>>> the current implementation. We are testing each combination of fields >>>>>> and >>>>>> options together. >>>>>> >>>>>> My current implementation is visible here: https://github.com/ >>>>>> PirosB3/django/compare/soc2014_meta_refactor >>>>>> >>>>>> For any questions or suggestions, let me know. >>>>>> >>>>>> Daniel Pyrathon >>>>>> >>>>>> >>>>>> On Monday, May 26, 2014 1:28:31 AM UTC+2, Daniel Pyrathon wrote: >>>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> Just to make you know, I have put up the current _meta API >>>>>>> documentation here: >>>>>>> http://162.219.6.191:8000/ref/models/meta.html?highlight=_meta >>>>>>> As always, feel free to ask questions. >>>>>>> >>>>>>> Daniel >>>>>>> >>>>>>> On Monday, May 26, 2014 1:26:27 AM UTC+2, Daniel Pyrathon wrote: >>>>>>>> >>>>>>>> Hi Josh, >>>>>>>> >>>>>>>> The meta API specified in the docs (https://github.com/PirosB3/ >>>>>>>> django/blob/meta_documentation/docs/ref/models/meta.txt) is the >>>>>>>> current API. I have documented this in order to understand more of the >>>>>>>> current implementation and it will be good to show a comparison when a >>>>>>>> new >>>>>>>> meta API will be approved. >>>>>>>> >>>>>>>> My current proposal (https://gist.github.com/ >>>>>>>> PirosB3/371704ed40ed093d5a82) and it will be discussed tomorrow >>>>>>>> with Russell. I will post as soon as I have updates. >>>>>>>> >>>>>>>> Daniel Pyrathon >>>>>>>> >>>>>>>> On Saturday, May 24, 2014 10:37:49 AM UTC+2, Josh Smeaton wrote: >>>>>>>>> >>>>>>>>> Hi Daniel, >>>>>>>>> >>>>>>>>> Nice work putting that document together. Is the meta document you >>>>>>>>> put together the current API or is it the API you are proposing? If >>>>>>>>> the >>>>>>>>> latter, a few suggestions (and if others disagree, please point that >>>>>>>>> out): >>>>>>>>> >>>>>>>>> - Remove all mention of caching. That should be an implementation >>>>>>>>> detail only, and not a requirement for other implementations. >>>>>>>>> - the *_with_model methods really rub me up the wrong way. I would >>>>>>>>> prefer always returning the _with_model variant, and letting the >>>>>>>>> caller >>>>>>>>> discard the model if they don't need it. >>>>>>>>> - I'm not a fan of virtual and concrete fields, though I have to >>>>>>>>> admit I'm not sure how they're different, especially in the context >>>>>>>>> of >>>>>>>>> different implementations. >>>>>>>>> - Not sure that m2m should be differentiated from related. >>>>>>>>> - init_name_map should be an implementation detail. >>>>>>>>> - normalize_together should be an implementation detail. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Josh >>>>>>>>> >>>>>>>>> On Saturday, 24 May 2014 05:05:02 UTC+10, Daniel Pyrathon wrote: >>>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> In the last days I have built a documentation of the current >>>>>>>>>> state of Options. Based on feedback and prototyping I have thought >>>>>>>>>> of a >>>>>>>>>> potential interface for _meta that can solve the issues currently >>>>>>>>>> present, >>>>>>>>>> such as redundancy (in code and in caching systems). The interface >>>>>>>>>> has also >>>>>>>>>> been thought to be maintainable and is a base that can be used to >>>>>>>>>> create >>>>>>>>>> custom meta stores. >>>>>>>>>> Obviously this is far from perfect, It will need many iterations >>>>>>>>>> and maybe it is too complex. I would really love to gain as much >>>>>>>>>> feedback >>>>>>>>>> as possible so it can be discussed with Russell and the community on >>>>>>>>>> Monday. >>>>>>>>>> >>>>>>>>>> The documentation of _meta can be found here: https://github.com/ >>>>>>>>>> PirosB3/django/blob/meta_documentation/docs/ref/models/meta.txt >>>>>>>>>> I will be refining the document in the next days, I will also be >>>>>>>>>> publishing the docs on a webserver and will be linking a URL soon. >>>>>>>>>> >>>>>>>>>> My proposal has been published here: >>>>>>>>>> https://gist.github.com/PirosB3/371704ed40ed093d5a82 >>>>>>>>>> In the next days I will be iterating over the feedback gained, >>>>>>>>>> and based on one very interesting suggestion on IRC, I will try to >>>>>>>>>> see how >>>>>>>>>> my API syntax looks in modelforms.py. >>>>>>>>>> >>>>>>>>>> As said previously, and feedback is greatly appreciated. >>>>>>>>>> >>>>>>>>>> Hi from Pycon IT! >>>>>>>>>> >>>>>>>>>> Daniel Pyrathon >>>>>>>>>> >>>>>>>>>> On Tuesday, May 20, 2014 3:25:45 PM UTC+2, Josh Smeaton wrote: >>>>>>>>>>> >>>>>>>>>>> Best of luck! >>>>>>>>>>> >>>>>>>>>>> On Tuesday, 20 May 2014 03:56:06 UTC+10, Daniel Pyrathon wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> Today I will be starting my weekly updates on my SoC project: >>>>>>>>>>>> refactoring Meta to a stable API. For anyone who missed out, you >>>>>>>>>>>> will be >>>>>>>>>>>> able to view it here: https://docs.google.com/document/d/1yp2_ >>>>>>>>>>>> skqkxyrc0egdRv6ofnRGCI9nmvxDFBkCXgy0Jwo/edit >>>>>>>>>>>> >>>>>>>>>>>> This week is the first official week of SoC. Me and my mentor >>>>>>>>>>>> (Russell) are initially approaching the work in the following way: >>>>>>>>>>>> >>>>>>>>>>>> - *Document the existing Meta API* >>>>>>>>>>>> For each endpoint, document the following: >>>>>>>>>>>> - Input parameters and return type >>>>>>>>>>>> - Caching pattern used >>>>>>>>>>>> - Where it's called from (internally and externally to >>>>>>>>>>>> Meta) >>>>>>>>>>>> - Why is it being called >>>>>>>>>>>> - When is it being called >>>>>>>>>>>> >>>>>>>>>>>> - *Propose an initial refactor plan* >>>>>>>>>>>> Once the documentation has been done, I should have a >>>>>>>>>>>> better idea of the current implementation. This will allow me >>>>>>>>>>>> to mock a >>>>>>>>>>>> proposed implementation that will be reviewed at my next update >>>>>>>>>>>> call, on >>>>>>>>>>>> Monday. >>>>>>>>>>>> >>>>>>>>>>>> My next update will be posted on Friday, just to make sure the >>>>>>>>>>>> community is informed of my progress. For any major updates that >>>>>>>>>>>> require >>>>>>>>>>>> community approval, I will be creating separate threads. >>>>>>>>>>>> My name on the internet is pirosb3, so if you want to have a >>>>>>>>>>>> chat about my progress feel free to contact me! The branch I am >>>>>>>>>>>> currently >>>>>>>>>>>> working on is https://github.com/PirosB3/ >>>>>>>>>>>> django/tree/meta_documentation >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Daniel Pyrathon >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> On Tuesday, May 20, 2014 3:25:45 PM UTC+2, Josh Smeaton wrote: >>>>>>>>>>> >>>>>>>>>>> Best of luck! >>>>>>>>>>> >>>>>>>>>>> On Tuesday, 20 May 2014 03:56:06 UTC+10, Daniel Pyrathon wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> Today I will be starting my weekly updates on my SoC project: >>>>>>>>>>>> refactoring Meta to a stable API. For anyone who missed out, you >>>>>>>>>>>> will be >>>>>>>>>>>> able to view it here: https://docs.google.com/document/d/1yp2_ >>>>>>>>>>>> skqkxyrc0egdRv6ofnRGCI9nmvxDFBkCXgy0Jwo/edit >>>>>>>>>>>> >>>>>>>>>>>> This week is the first official week of SoC. Me and my mentor >>>>>>>>>>>> (Russell) are initially approaching the work in the following way: >>>>>>>>>>>> >>>>>>>>>>>> - *Document the existing Meta API* >>>>>>>>>>>> For each endpoint, document the following: >>>>>>>>>>>> - Input parameters and return type >>>>>>>>>>>> - Caching pattern used >>>>>>>>>>>> - Where it's called from (internally and externally to >>>>>>>>>>>> Meta) >>>>>>>>>>>> - Why is it being called >>>>>>>>>>>> - When is it being called >>>>>>>>>>>> >>>>>>>>>>>> - *Propose an initial refactor plan* >>>>>>>>>>>> Once the documentation has been done, I should have a >>>>>>>>>>>> better idea of the current implementation. This will allow me >>>>>>>>>>>> to mock a >>>>>>>>>>>> proposed implementation that will be reviewed at my next update >>>>>>>>>>>> call, on >>>>>>>>>>>> Monday. >>>>>>>>>>>> >>>>>>>>>>>> My next update will be posted on Friday, just to make sure the >>>>>>>>>>>> community is informed of my progress. For any major updates that >>>>>>>>>>>> require >>>>>>>>>>>> community approval, I will be creating separate threads. >>>>>>>>>>>> My name on the internet is pirosb3, so if you want to have a >>>>>>>>>>>> chat about my progress feel free to contact me! The branch I am >>>>>>>>>>>> currently >>>>>>>>>>>> working on is https://github.com/PirosB3/ >>>>>>>>>>>> django/tree/meta_documentation >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Daniel Pyrathon >>>>>>>>>>>> >>>>>>>>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Django developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> >>>> Visit this group at http://groups.google.com/group/django-developers. >>>> To view this discussion on the web visit https://groups.google. >>>> com/d/msgid/django-developers/78f45c61-9626-4aa9-a854- >>>> 64b11ba44572%40googlegroups.com >>>> <https://groups.google.com/d/msgid/django-developers/78f45c61-9626-4aa9-a854-64b11ba44572%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/django-developers. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/django-developers/445fea1d-406c-4ae9-b869-c02f189a2b86%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/django-developers/445fea1d-406c-4ae9-b869-c02f189a2b86%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d87c6ed4-9bf0-4c6f-8b3a-0a3ea6b7dae1%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
