Did the "custom_qualifier" patch make it into root? I can see it is used in dal/pydal/adapters/google_adapters.py but with any values passed straight through it's not easy to work out what is/isn't supported.
On Sunday, 10 March 2013 02:51:11 UTC, Christian Foster Howes wrote: > > Massimo, Scott, > > i filed https://code.google.com/p/web2py/issues/detail?id=1373 with a > patch against trunk. Looks like my first patch on this thread made it to > the latest release, but the second patch did not. If that can be applied i > think we'll be all set. > > perhaps i now need to start submitting book patches to write up these > features! > > cfh > > On Thursday, February 28, 2013 9:35:33 AM UTC-8, Scott Hunter wrote: >> >> Sorry, I thought I had -- it works like a champ. Would be nice if there >> were a way to mark the signature fields as unindexed w/o having to recreate >> them (not as simple as cut and paste, as the table def uses a bunch of >> local definitions, but not hard), but definitely workable. And lowers the >> index overhead a LOT; in my case, from 15K to <3K. >> >> - Scott >> >> On Thursday, February 28, 2013 11:12:52 AM UTC-5, howesc wrote: >>> >>> can you let me know if it works as desired now? if so, i'll submit the >>> patch to massimo for review and possible inclusion in the next release of >>> web2py. >>> >>> thanks, >>> >>> cfh >>> >>> On Tuesday, February 26, 2013 6:31:52 PM UTC-8, Scott Hunter wrote: >>>> >>>> With the latest patch, I was able to disable indices on the string >>>> fields I wanted to. Unfortunately (for me), I've just about used up my >>>> quota, so I'll have to wait until tomorrow to see how much doing so saves >>>> me. >>>> >>>> Thanks, >>>> Scott >>>> >>>> On Tuesday, February 26, 2013 8:50:29 PM UTC-5, howesc wrote: >>>>> >>>>> here's an updated DAL patch to try..... >>>>> >>>>> thanks for trudging through this with us! >>>>> >>>>> cfh >>>>> >>>>> On Tuesday, February 26, 2013 3:36:30 PM UTC-8, Scott Hunter wrote: >>>>>> >>>>>> I'm working from Version 2.3.2 (2012-12-17 15:03:30) stable >>>>>> >>>>>> Here's the table in question (I've commented-out the custom_qualifier >>>>>> for the string fields): >>>>>> >>>>>> db.define_table('t_run', >>>>>> Field('f_trial', type='reference t_trial', >>>>>> label=T('Trial')), >>>>>> Field('f_when', type='date', >>>>>> label=T('When')), >>>>>> Field('f_dog', type='reference t_dog', >>>>>> label=T('Dog')), >>>>>> Field('f_name', type='string', >>>>>> #custom_qualifier={'indexed':False}, >>>>>> label=T('Name')), >>>>>> Field('f_breed', type='string', >>>>>> #custom_qualifier={'indexed':False}, >>>>>> label=T('Breed')), >>>>>> Field('f_jump_height', type='integer',custom_qualifier={'indexed' >>>>>> :False}, >>>>>> label=T('Jump Height')), >>>>>> Field('f_level', type='string', >>>>>> #custom_qualifier={'indexed':False}, >>>>>> label=T('Level')), >>>>>> Field('f_class', type='string', >>>>>> #custom_qualifier={'indexed':False}, >>>>>> label=T('Class')), >>>>>> Field('f_pref', type='boolean',custom_qualifier={'indexed':False >>>>>> }, >>>>>> label=T('Preferred')), >>>>>> Field('f_armband', type='string', >>>>>> #custom_qualifier={'indexed':False}, >>>>>> label=T('Armband')), >>>>>> Field('f_yards', type='integer',custom_qualifier={'indexed':False >>>>>> }, >>>>>> label=T('Yards')), >>>>>> Field('f_sct', type='integer',custom_qualifier={'indexed':False}, >>>>>> label=T('SCT')), >>>>>> Field('f_judge', type='string', >>>>>> #custom_qualifier={'indexed':False}, >>>>>> label=T('Judge')), >>>>>> Field('f_score', type='integer',custom_qualifier={'indexed':False >>>>>> }, >>>>>> label=T('Score')), >>>>>> Field('f_time', type='double',custom_qualifier={'indexed':False}, >>>>>> label=T('Time')), >>>>>> Field('f_faults', type='string', >>>>>> #custom_qualifier={'indexed':False}, >>>>>> label=T('Faults')), >>>>>> Field('f_handler', type='string', >>>>>> #custom_qualifier={'indexed':False}, >>>>>> label=T('Handler')), >>>>>> Field('f_order', type='integer', >>>>>> label=T('Order')), >>>>>> auth.signature, >>>>>> format='%(f_key)s', >>>>>> migrate=settings.migrate) >>>>>> >>>>>> >>>>>> P.S. Turning off the indexes DOES make a difference; with the changes >>>>>> above, instead of using up over 30% of my quota, I'm "only" using 23%. >>>>>> If >>>>>> I can get the strings unindexed, and take out the web2py-supplied fields >>>>>> (created_by & _on, modified_by & _on, and maybe is_active), that should >>>>>> get >>>>>> it down to something manageable. (Not clear on how to handle the >>>>>> web2py-supplied fields, as I don't know what parameters were using in >>>>>> making them, making it difficult to know just how to "supply my own >>>>>> definitions".) >>>>>> >>>>>> On Tuesday, February 26, 2013 6:24:21 PM UTC-5, howesc wrote: >>>>>>> >>>>>>> your line numbers are off from mine so i'm having trouble making >>>>>>> sense of this. :( >>>>>>> >>>>>>> can you send your model definition so i can see what you are working >>>>>>> with? i think then i can line it up with the dal version i have here >>>>>>> (which was trunk from HG as of saturday AM PST) >>>>>>> >>>>>>> cfh >>>>>>> >>>>>>> On Tuesday, February 26, 2013 3:18:13 PM UTC-8, Scott Hunter wrote: >>>>>>>> >>>>>>>> Sorry about that. I've fixed it, and now I get the following: >>>>>>>> >>>>>>>> In FILE: /base/data/home/apps/s~sbhweb2py/1.365574604253984974/ >>>>>>>> applications/ppt_demo/models/db_wizard.py >>>>>>>> >>>>>>>> >>>>>>>> Traceback (most recent call last): >>>>>>>> File >>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/restricted.py" >>>>>>>> , line 212, in restricted >>>>>>>> exec ccode in environment >>>>>>>> File >>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/applications/ppt_demo/models/db_wizard.py" >>>>>>>> , line 198, in <module> >>>>>>>> migrate=settings.migrate) >>>>>>>> File >>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/dal.py" >>>>>>>> , line 7189, in define_table >>>>>>>> table = self.lazy_define_table(tablename,*fields,**args) >>>>>>>> File >>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/dal.py" >>>>>>>> , line 7225, in lazy_define_table >>>>>>>> polymodel=polymodel) >>>>>>>> File >>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/dal.py" >>>>>>>> , line 4368, in create_table >>>>>>>> ftype = self.types[field_type](**attr) >>>>>>>> TypeError: <lambda>() takes no arguments (1 given) >>>>>>>> >>>>>>>> >>>>>>>> This only happens if one of the fields in question is a string; I >>>>>>>> get no error when the unindexed fields are all integer, boolean or >>>>>>>> double. >>>>>>>> >>>>>>>> - Scott >>>>>>>> >>>>>>>> >>>>>>>> On Tuesday, February 26, 2013 1:04:12 AM UTC-5, howesc wrote: >>>>>>>>> >>>>>>>>> it looks like you have a typo.... "custom_qualifer" vs >>>>>>>>> "custom_qualifier" >>>>>>>>> >>>>>>>>> On Monday, February 25, 2013 6:44:14 PM UTC-8, Scott Hunter wrote: >>>>>>>>>> >>>>>>>>>> I applied the patch, and added custom_qualifiers like so: >>>>>>>>>> >>>>>>>>>> Field('f_name', type='string',custom_qualifer={'indexed': >>>>>>>>>> False}, >>>>>>>>>> label=T('Name')), >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> and this is the error I got: >>>>>>>>>> >>>>>>>>>> In FILE: /base/data/home/apps/s~sbhweb2py/1.365567821359373728/ >>>>>>>>>> applications/ppt_demo/models/db_wizard.py >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Traceback (most recent call last): >>>>>>>>>> File >>>>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365567821359373728/gluon/restricted.py" >>>>>>>>>> , line 212, in restricted >>>>>>>>>> exec ccode in environment >>>>>>>>>> File >>>>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365567821359373728/applications/ppt_demo/models/db_wizard.py" >>>>>>>>>> , line 165, in <module> >>>>>>>>>> label=T('Name')), >>>>>>>>>> TypeError: __init__() got an unexpected keyword argument >>>>>>>>>> 'custom_qualifer' >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Saturday, February 23, 2013 12:30:48 PM UTC-5, howesc wrote: >>>>>>>>>>> >>>>>>>>>>> Scott, >>>>>>>>>>> >>>>>>>>>>> this is *completely* untested, but here's a proposal: >>>>>>>>>>> - use the (undocumented) field.custom_qualifier property in GAE >>>>>>>>>>> field definitions >>>>>>>>>>> - if you want a field to be unindexed set >>>>>>>>>>> custom_qualifier={'indexed':False} >>>>>>>>>>> - for "properties that don't get listed explicitly"....override >>>>>>>>>>> those default properties with what you want (there are a few >>>>>>>>>>> techniques for >>>>>>>>>>> not explicitly listing fields, depending on which technique you are >>>>>>>>>>> using >>>>>>>>>>> the answer here is different. >>>>>>>>>>> >>>>>>>>>>> this patch against HG trunk this AM might work (i say might >>>>>>>>>>> cause i have not tested it). are you willing to experiment with it >>>>>>>>>>> and let >>>>>>>>>>> us know? >>>>>>>>>>> >>>>>>>>>>> thanks, >>>>>>>>>>> >>>>>>>>>>> christian >>>>>>>>>>> >>>>>>>>>>> On Friday, February 22, 2013 3:53:33 PM UTC-8, Scott Hunter >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> If I had to guess, I'd say a patch is needed in the loop over >>>>>>>>>>>> the fields in create_table, that would add a new entry to the dict >>>>>>>>>>>> for the >>>>>>>>>>>> field definition based on a new attribute (which would be ignored >>>>>>>>>>>> for >>>>>>>>>>>> anything other than Google's Datastore) which gets added to >>>>>>>>>>>> sql_fields; >>>>>>>>>>>> then migrate_table, which seems to build the table >>>>>>>>>>>> building/altering >>>>>>>>>>>> commands would need to be made to recognize the new entry & add >>>>>>>>>>>> the disable >>>>>>>>>>>> index command. There's not a lot of comments in the code, and I'd >>>>>>>>>>>> be >>>>>>>>>>>> afraid to break something, but might give it a try. >>>>>>>>>>>> >>>>>>>>>>>> - Scott >>>>>>>>>>>> >>>>>>>>>>>> P.S. Not sure of the best way to handle this (what's describe >>>>>>>>>>>> above wouldn't handle it), but it would be nice to be able to turn >>>>>>>>>>>> off the >>>>>>>>>>>> indexes for the fields that don't get listed explicitly in >>>>>>>>>>>> models.db >>>>>>>>>>>> (things like created by & when, modified by & when). >>>>>>>>>>>> >>>>>>>>>>>> On Friday, February 22, 2013 6:14:28 PM UTC-5, howesc wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> thanks for the link. i'll try and take a look this weekend >>>>>>>>>>>>> and see if there is a place for that in the DAL (feel free to >>>>>>>>>>>>> open >>>>>>>>>>>>> gluon/dal.py yourself too). >>>>>>>>>>>>> >>>>>>>>>>>>> yup, i'm paying more for writes right now then i am for >>>>>>>>>>>>> instance hours per day on my largest paid application. :) >>>>>>>>>>>>> >>>>>>>>>>>>> cfh >>>>>>>>>>>>> >>>>>>>>>>>>> On Thursday, February 21, 2013 5:59:06 PM UTC-8, Scott Hunter >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> According to >>>>>>>>>>>>>> https://developers.google.com/appengine/docs/python/datastore/indexes#Unindexed_Properties, >>>>>>>>>>>>>> >>>>>>>>>>>>>> "You declare a property unindexed by setting indexed=False in >>>>>>>>>>>>>> the property >>>>>>>>>>>>>> constructor". >>>>>>>>>>>>>> >>>>>>>>>>>>>> One incurs the cost of a write (or 2?) for every property of >>>>>>>>>>>>>> a record that gets written; so, for a table with about 25 fields >>>>>>>>>>>>>> (once you >>>>>>>>>>>>>> include all of the ones web2py adds), inserting 300 records >>>>>>>>>>>>>> incurs the cost >>>>>>>>>>>>>> of 14K+ writes; as the free account has a limit of 50K writes >>>>>>>>>>>>>> per day, that >>>>>>>>>>>>>> is rather limiting. It is easy to turn such indicies back on, >>>>>>>>>>>>>> but you have >>>>>>>>>>>>>> to re-write every record in order to repopulate them. >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Scott >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thursday, February 21, 2013 4:51:59 PM UTC-5, howesc wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> i'm being lazy here....do you have the instructions (or link >>>>>>>>>>>>>>> to instructions) from GAE for disabling indexes? i don't have >>>>>>>>>>>>>>> it handy >>>>>>>>>>>>>>> right now.... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> we can check if the DAL has a secret way to handle it, >>>>>>>>>>>>>>> and/or create a patch to allow it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> may i ask what advantage you are hoping to achieve by >>>>>>>>>>>>>>> skipping those indexes? i've left them on for fear of making a >>>>>>>>>>>>>>> bad choice >>>>>>>>>>>>>>> that i can't revert. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> christian >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thursday, February 21, 2013 5:56:07 AM UTC-8, Scott >>>>>>>>>>>>>>> Hunter wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> As I understand it, the GAE datastore will automatically >>>>>>>>>>>>>>>> create an index for every field of every table (to facilitate >>>>>>>>>>>>>>>> queries using >>>>>>>>>>>>>>>> single fields); those requiring multiple fields also get >>>>>>>>>>>>>>>> generated, but >>>>>>>>>>>>>>>> will get added to index.yaml by the SDK when tested locally. >>>>>>>>>>>>>>>> Since these >>>>>>>>>>>>>>>> are created outside of the DAL, I wouldn't expect to be able >>>>>>>>>>>>>>>> to remove them >>>>>>>>>>>>>>>> using it. I also understand that there is a way, with the GAE >>>>>>>>>>>>>>>> SDK, to >>>>>>>>>>>>>>>> disable the index for any given field (by a parameter in the >>>>>>>>>>>>>>>> field >>>>>>>>>>>>>>>> definition); is there any way to accomplish this through the >>>>>>>>>>>>>>>> DAL, and if >>>>>>>>>>>>>>>> so, what is it? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
