Public bug reported: When performing bulk actions, one area that is still very much forced into the singleton method of processing is the resource_extend framework, where extensions can register to extend object data. This RFE is to propose the changes needed to make resource_extend work in a bulk context.
The details of the proposal are as follows: 1.) Add an optional additional argument "bulk" to @resource_extend.extends that would default to False. If it is True this signals that the function being decorated can support bulk extensions. Normally the arguments to such a function are in the form (resource, db_model); if it is operating in bulk mode then the first argument would be treated as [resource, model] in an array, and the second argument would be ignored. This can be tested for inside the function by checking if the first argument is an array. 2.) Create a new resource_extend.apply_funcs_bulk function that would act like the current resource_extend.apply_funcs but would take an array of object data instead of a single object resource and db model object. This new function would consult the "bulk" attribute on registered extender functions and would pass in the array of object data to a bulk function, but if the function was bulk=False then it would loop through as it does now. 3.) This would require a revamp of the _resource_extend_functions data structure in neutron-lib/db/resource_extends.py. Currently it is a dict with the resource type as a string key, and the value is an array of functions to be applied. This would need to be changed to take advantage of the bulk attribute. I am not sure how best to do that at this point. One scenario where this would help is with bulk port creation, because resource extender functions could implement more efficient commits of their various data. Another example is a bug from when I was last bug deputy [1], where a user is complaining of slow responses due to large quantities of SQL queries when using large numbers of trunk ports. I think that with a bulk resource extend the trunk port extender could be implemented as qa single SQL query that fetches information for all affected ports, thus significantly increasing performance. Please let me know what you think of this idea. I raised it as an RFE because it requires significant enough modification to an element of Neutron that is both foundational and that not many people may have dealt with directly that an open discussion of the merits and implementation would be a positive contribution. [1] https://bugs.launchpad.net/neutron/+bug/1842150 ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1843924 Title: [RFE] Create optional bulk resource_extend Status in neutron: New Bug description: When performing bulk actions, one area that is still very much forced into the singleton method of processing is the resource_extend framework, where extensions can register to extend object data. This RFE is to propose the changes needed to make resource_extend work in a bulk context. The details of the proposal are as follows: 1.) Add an optional additional argument "bulk" to @resource_extend.extends that would default to False. If it is True this signals that the function being decorated can support bulk extensions. Normally the arguments to such a function are in the form (resource, db_model); if it is operating in bulk mode then the first argument would be treated as [resource, model] in an array, and the second argument would be ignored. This can be tested for inside the function by checking if the first argument is an array. 2.) Create a new resource_extend.apply_funcs_bulk function that would act like the current resource_extend.apply_funcs but would take an array of object data instead of a single object resource and db model object. This new function would consult the "bulk" attribute on registered extender functions and would pass in the array of object data to a bulk function, but if the function was bulk=False then it would loop through as it does now. 3.) This would require a revamp of the _resource_extend_functions data structure in neutron-lib/db/resource_extends.py. Currently it is a dict with the resource type as a string key, and the value is an array of functions to be applied. This would need to be changed to take advantage of the bulk attribute. I am not sure how best to do that at this point. One scenario where this would help is with bulk port creation, because resource extender functions could implement more efficient commits of their various data. Another example is a bug from when I was last bug deputy [1], where a user is complaining of slow responses due to large quantities of SQL queries when using large numbers of trunk ports. I think that with a bulk resource extend the trunk port extender could be implemented as qa single SQL query that fetches information for all affected ports, thus significantly increasing performance. Please let me know what you think of this idea. I raised it as an RFE because it requires significant enough modification to an element of Neutron that is both foundational and that not many people may have dealt with directly that an open discussion of the merits and implementation would be a positive contribution. [1] https://bugs.launchpad.net/neutron/+bug/1842150 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1843924/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
