Massimo, Scott,
i filed https://code.google.com/p/web2py/issues/detail?id=1373 with a patch
against trunk. Looks like my first patch on this thread made it to the
latest release, but the second patch did not. If that can be applied i
think we'll be all set.
perhaps i now need to start submitting book patches to write up these
features!
cfh
On Thursday, February 28, 2013 9:35:33 AM UTC-8, Scott Hunter wrote:
>
> Sorry, I thought I had -- it works like a champ. Would be nice if there
> were a way to mark the signature fields as unindexed w/o having to recreate
> them (not as simple as cut and paste, as the table def uses a bunch of
> local definitions, but not hard), but definitely workable. And lowers the
> index overhead a LOT; in my case, from 15K to <3K.
>
> - Scott
>
> On Thursday, February 28, 2013 11:12:52 AM UTC-5, howesc wrote:
>>
>> can you let me know if it works as desired now? if so, i'll submit the
>> patch to massimo for review and possible inclusion in the next release of
>> web2py.
>>
>> thanks,
>>
>> cfh
>>
>> On Tuesday, February 26, 2013 6:31:52 PM UTC-8, Scott Hunter wrote:
>>>
>>> With the latest patch, I was able to disable indices on the string
>>> fields I wanted to. Unfortunately (for me), I've just about used up my
>>> quota, so I'll have to wait until tomorrow to see how much doing so saves
>>> me.
>>>
>>> Thanks,
>>> Scott
>>>
>>> On Tuesday, February 26, 2013 8:50:29 PM UTC-5, howesc wrote:
>>>>
>>>> here's an updated DAL patch to try.....
>>>>
>>>> thanks for trudging through this with us!
>>>>
>>>> cfh
>>>>
>>>> On Tuesday, February 26, 2013 3:36:30 PM UTC-8, Scott Hunter wrote:
>>>>>
>>>>> I'm working from Version 2.3.2 (2012-12-17 15:03:30) stable
>>>>>
>>>>> Here's the table in question (I've commented-out the custom_qualifier
>>>>> for the string fields):
>>>>>
>>>>> db.define_table('t_run',
>>>>> Field('f_trial', type='reference t_trial',
>>>>> label=T('Trial')),
>>>>> Field('f_when', type='date',
>>>>> label=T('When')),
>>>>> Field('f_dog', type='reference t_dog',
>>>>> label=T('Dog')),
>>>>> Field('f_name', type='string',#custom_qualifier={'indexed':False},
>>>>> label=T('Name')),
>>>>> Field('f_breed', type='string',
>>>>> #custom_qualifier={'indexed':False},
>>>>> label=T('Breed')),
>>>>> Field('f_jump_height', type='integer',custom_qualifier={'indexed':
>>>>> False},
>>>>> label=T('Jump Height')),
>>>>> Field('f_level', type='string',
>>>>> #custom_qualifier={'indexed':False},
>>>>> label=T('Level')),
>>>>> Field('f_class', type='string',
>>>>> #custom_qualifier={'indexed':False},
>>>>> label=T('Class')),
>>>>> Field('f_pref', type='boolean',custom_qualifier={'indexed':False},
>>>>> label=T('Preferred')),
>>>>> Field('f_armband', type='string',
>>>>> #custom_qualifier={'indexed':False},
>>>>> label=T('Armband')),
>>>>> Field('f_yards', type='integer',custom_qualifier={'indexed':False
>>>>> },
>>>>> label=T('Yards')),
>>>>> Field('f_sct', type='integer',custom_qualifier={'indexed':False},
>>>>> label=T('SCT')),
>>>>> Field('f_judge', type='string',
>>>>> #custom_qualifier={'indexed':False},
>>>>> label=T('Judge')),
>>>>> Field('f_score', type='integer',custom_qualifier={'indexed':False
>>>>> },
>>>>> label=T('Score')),
>>>>> Field('f_time', type='double',custom_qualifier={'indexed':False},
>>>>> label=T('Time')),
>>>>> Field('f_faults', type='string',
>>>>> #custom_qualifier={'indexed':False},
>>>>> label=T('Faults')),
>>>>> Field('f_handler', type='string',
>>>>> #custom_qualifier={'indexed':False},
>>>>> label=T('Handler')),
>>>>> Field('f_order', type='integer',
>>>>> label=T('Order')),
>>>>> auth.signature,
>>>>> format='%(f_key)s',
>>>>> migrate=settings.migrate)
>>>>>
>>>>>
>>>>> P.S. Turning off the indexes DOES make a difference; with the changes
>>>>> above, instead of using up over 30% of my quota, I'm "only" using 23%.
>>>>> If
>>>>> I can get the strings unindexed, and take out the web2py-supplied fields
>>>>> (created_by & _on, modified_by & _on, and maybe is_active), that should
>>>>> get
>>>>> it down to something manageable. (Not clear on how to handle the
>>>>> web2py-supplied fields, as I don't know what parameters were using in
>>>>> making them, making it difficult to know just how to "supply my own
>>>>> definitions".)
>>>>>
>>>>> On Tuesday, February 26, 2013 6:24:21 PM UTC-5, howesc wrote:
>>>>>>
>>>>>> your line numbers are off from mine so i'm having trouble making
>>>>>> sense of this. :(
>>>>>>
>>>>>> can you send your model definition so i can see what you are working
>>>>>> with? i think then i can line it up with the dal version i have here
>>>>>> (which was trunk from HG as of saturday AM PST)
>>>>>>
>>>>>> cfh
>>>>>>
>>>>>> On Tuesday, February 26, 2013 3:18:13 PM UTC-8, Scott Hunter wrote:
>>>>>>>
>>>>>>> Sorry about that. I've fixed it, and now I get the following:
>>>>>>>
>>>>>>> In FILE: /base/data/home/apps/s~sbhweb2py/1.365574604253984974/
>>>>>>> applications/ppt_demo/models/db_wizard.py
>>>>>>>
>>>>>>>
>>>>>>> Traceback (most recent call last):
>>>>>>> File
>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/restricted.py"
>>>>>>> , line 212, in restricted
>>>>>>> exec ccode in environment
>>>>>>> File
>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/applications/ppt_demo/models/db_wizard.py"
>>>>>>> , line 198, in <module>
>>>>>>> migrate=settings.migrate)
>>>>>>> File
>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/dal.py"
>>>>>>> , line 7189, in define_table
>>>>>>> table = self.lazy_define_table(tablename,*fields,**args)
>>>>>>> File
>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/dal.py"
>>>>>>> , line 7225, in lazy_define_table
>>>>>>> polymodel=polymodel)
>>>>>>> File
>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/dal.py"
>>>>>>> , line 4368, in create_table
>>>>>>> ftype = self.types[field_type](**attr)
>>>>>>> TypeError: <lambda>() takes no arguments (1 given)
>>>>>>>
>>>>>>>
>>>>>>> This only happens if one of the fields in question is a string; I
>>>>>>> get no error when the unindexed fields are all integer, boolean or
>>>>>>> double.
>>>>>>>
>>>>>>> - Scott
>>>>>>>
>>>>>>>
>>>>>>> On Tuesday, February 26, 2013 1:04:12 AM UTC-5, howesc wrote:
>>>>>>>>
>>>>>>>> it looks like you have a typo.... "custom_qualifer" vs
>>>>>>>> "custom_qualifier"
>>>>>>>>
>>>>>>>> On Monday, February 25, 2013 6:44:14 PM UTC-8, Scott Hunter wrote:
>>>>>>>>>
>>>>>>>>> I applied the patch, and added custom_qualifiers like so:
>>>>>>>>>
>>>>>>>>> Field('f_name', type='string',custom_qualifer={'indexed':False
>>>>>>>>> },
>>>>>>>>> label=T('Name')),
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> and this is the error I got:
>>>>>>>>>
>>>>>>>>> In FILE: /base/data/home/apps/s~sbhweb2py/1.365567821359373728/
>>>>>>>>> applications/ppt_demo/models/db_wizard.py
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>> File
>>>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365567821359373728/gluon/restricted.py"
>>>>>>>>> , line 212, in restricted
>>>>>>>>> exec ccode in environment
>>>>>>>>> File
>>>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365567821359373728/applications/ppt_demo/models/db_wizard.py"
>>>>>>>>> , line 165, in <module>
>>>>>>>>> label=T('Name')),
>>>>>>>>> TypeError: __init__() got an unexpected keyword argument
>>>>>>>>> 'custom_qualifer'
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Saturday, February 23, 2013 12:30:48 PM UTC-5, howesc wrote:
>>>>>>>>>>
>>>>>>>>>> Scott,
>>>>>>>>>>
>>>>>>>>>> this is *completely* untested, but here's a proposal:
>>>>>>>>>> - use the (undocumented) field.custom_qualifier property in GAE
>>>>>>>>>> field definitions
>>>>>>>>>> - if you want a field to be unindexed set
>>>>>>>>>> custom_qualifier={'indexed':False}
>>>>>>>>>> - for "properties that don't get listed explicitly"....override
>>>>>>>>>> those default properties with what you want (there are a few
>>>>>>>>>> techniques for
>>>>>>>>>> not explicitly listing fields, depending on which technique you are
>>>>>>>>>> using
>>>>>>>>>> the answer here is different.
>>>>>>>>>>
>>>>>>>>>> this patch against HG trunk this AM might work (i say might cause
>>>>>>>>>> i have not tested it). are you willing to experiment with it and
>>>>>>>>>> let us
>>>>>>>>>> know?
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>>
>>>>>>>>>> christian
>>>>>>>>>>
>>>>>>>>>> On Friday, February 22, 2013 3:53:33 PM UTC-8, Scott Hunter wrote:
>>>>>>>>>>>
>>>>>>>>>>> If I had to guess, I'd say a patch is needed in the loop over
>>>>>>>>>>> the fields in create_table, that would add a new entry to the dict
>>>>>>>>>>> for the
>>>>>>>>>>> field definition based on a new attribute (which would be ignored
>>>>>>>>>>> for
>>>>>>>>>>> anything other than Google's Datastore) which gets added to
>>>>>>>>>>> sql_fields;
>>>>>>>>>>> then migrate_table, which seems to build the table
>>>>>>>>>>> building/altering
>>>>>>>>>>> commands would need to be made to recognize the new entry & add the
>>>>>>>>>>> disable
>>>>>>>>>>> index command. There's not a lot of comments in the code, and I'd
>>>>>>>>>>> be
>>>>>>>>>>> afraid to break something, but might give it a try.
>>>>>>>>>>>
>>>>>>>>>>> - Scott
>>>>>>>>>>>
>>>>>>>>>>> P.S. Not sure of the best way to handle this (what's describe
>>>>>>>>>>> above wouldn't handle it), but it would be nice to be able to turn
>>>>>>>>>>> off the
>>>>>>>>>>> indexes for the fields that don't get listed explicitly in
>>>>>>>>>>> models.db
>>>>>>>>>>> (things like created by & when, modified by & when).
>>>>>>>>>>>
>>>>>>>>>>> On Friday, February 22, 2013 6:14:28 PM UTC-5, howesc wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> thanks for the link. i'll try and take a look this weekend and
>>>>>>>>>>>> see if there is a place for that in the DAL (feel free to open
>>>>>>>>>>>> gluon/dal.py yourself too).
>>>>>>>>>>>>
>>>>>>>>>>>> yup, i'm paying more for writes right now then i am for
>>>>>>>>>>>> instance hours per day on my largest paid application. :)
>>>>>>>>>>>>
>>>>>>>>>>>> cfh
>>>>>>>>>>>>
>>>>>>>>>>>> On Thursday, February 21, 2013 5:59:06 PM UTC-8, Scott Hunter
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> According to
>>>>>>>>>>>>> https://developers.google.com/appengine/docs/python/datastore/indexes#Unindexed_Properties,
>>>>>>>>>>>>>
>>>>>>>>>>>>> "You declare a property unindexed by setting indexed=False in the
>>>>>>>>>>>>> property
>>>>>>>>>>>>> constructor".
>>>>>>>>>>>>>
>>>>>>>>>>>>> One incurs the cost of a write (or 2?) for every property of a
>>>>>>>>>>>>> record that gets written; so, for a table with about 25 fields
>>>>>>>>>>>>> (once you
>>>>>>>>>>>>> include all of the ones web2py adds), inserting 300 records
>>>>>>>>>>>>> incurs the cost
>>>>>>>>>>>>> of 14K+ writes; as the free account has a limit of 50K writes per
>>>>>>>>>>>>> day, that
>>>>>>>>>>>>> is rather limiting. It is easy to turn such indicies back on,
>>>>>>>>>>>>> but you have
>>>>>>>>>>>>> to re-write every record in order to repopulate them.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Scott
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thursday, February 21, 2013 4:51:59 PM UTC-5, howesc wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> i'm being lazy here....do you have the instructions (or link
>>>>>>>>>>>>>> to instructions) from GAE for disabling indexes? i don't have
>>>>>>>>>>>>>> it handy
>>>>>>>>>>>>>> right now....
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> we can check if the DAL has a secret way to handle it, and/or
>>>>>>>>>>>>>> create a patch to allow it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> may i ask what advantage you are hoping to achieve by
>>>>>>>>>>>>>> skipping those indexes? i've left them on for fear of making a
>>>>>>>>>>>>>> bad choice
>>>>>>>>>>>>>> that i can't revert.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> christian
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thursday, February 21, 2013 5:56:07 AM UTC-8, Scott Hunter
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As I understand it, the GAE datastore will automatically
>>>>>>>>>>>>>>> create an index for every field of every table (to facilitate
>>>>>>>>>>>>>>> queries using
>>>>>>>>>>>>>>> single fields); those requiring multiple fields also get
>>>>>>>>>>>>>>> generated, but
>>>>>>>>>>>>>>> will get added to index.yaml by the SDK when tested locally.
>>>>>>>>>>>>>>> Since these
>>>>>>>>>>>>>>> are created outside of the DAL, I wouldn't expect to be able to
>>>>>>>>>>>>>>> remove them
>>>>>>>>>>>>>>> using it. I also understand that there is a way, with the GAE
>>>>>>>>>>>>>>> SDK, to
>>>>>>>>>>>>>>> disable the index for any given field (by a parameter in the
>>>>>>>>>>>>>>> field
>>>>>>>>>>>>>>> definition); is there any way to accomplish this through the
>>>>>>>>>>>>>>> DAL, and if
>>>>>>>>>>>>>>> so, what is it?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
--
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
