Indeed... :) Ok, next question for you/all...
How could I differentiate between what is a resource creation and a resource modification? Have updated the netapp_user provider to prefetch/flush, however am struggling to differentiate between what is a resource creation, and what is a modification... Code is here: https://github.com/fatmcgav/fatmcgav-netapp/commit/66092978f4182c5474a60011db99ee2e3e12e689 Thoughts? Cheers Gavin On Tuesday, 2 April 2013 16:17:33 UTC+1, Luke Kanies wrote: > > That is awesome. It's amazing what a bit of optimization will do for you. > > On Apr 2, 2013, at 3:12 PM, fatmcgav <[email protected] <javascript:>> > wrote: > > Yep, have been testing it today on my dev and production Puppet masters, > and seems to be working as expected... > > Have been very impressed with the performance improvements - Has dropped > the netapp_volume resource time from 40+ seconds to <0.1 seconds. So > MASSIVE improvements to be made from switching to the prefetch/flush > style... > > Cheers > Gav > > > On 2 April 2013 13:24, Luke Kanies <[email protected] <javascript:>>wrote: > >> Looks good from a cursory glance. Hopefully it's all working for you. >> >> On Apr 2, 2013, at 9:52 AM, Gavin Williams <[email protected]<javascript:>> >> wrote: >> >> As a quick update, I've pushed my latest set of code tweaks up - >> https://github.com/fatmcgav/fatmcgav-netapp/commit/7a7d18bf39cdbb04a3b0b5192929ec6a857c0a5e >> >> Need to tidy the code up a bit, but should be able to get the gist :) >> Have got a couple of defs which pull back the bulk of the information, and >> then a couple that need to be called on a volume by volume basis. >> All gets put together in instances, and provider'd up in prefetch. >> >> Comments welcome. >> >> Cheers >> Gavin >> >> On Friday, 29 March 2013 13:15:12 UTC, Gavin Williams wrote: >>> >>> Luke >>> >>> Cheers for that info, should prive useful... >>> >>> Guess I could implement a method for each property that pulls back all >>> that property values for all the volumes, and then match them up in >>> prefetch. >>> >>> Cheers >>> Gav >>> On 29 Mar 2013 12:49, "Luke Kanies" <[email protected] <javascript:>> >>> wrote: >>> >>>> Hi Gavin, >>>> >>>> Yeah, there isn't as much consistency in API as I would like. >>>> >>>> The way it should work, I believe, is something like: >>>> >>>> * The Type should support an 'instances' method that returns a complete >>>> list of all instances of that type that exist on the host itself. This >>>> method should not require a catalog. >>>> >>>> * The provider should support a 'prefetch' method, which accepts a >>>> list of resources from the catalog and updates them with the correct >>>> provider instance, with appropriate data already filled in to reflect the >>>> system state. >>>> >>>> * Optionally, many providers implement (I think) their own 'instances' >>>> method, which returns all instances and doesn't require a catalog, to >>>> implement both of the above methods. >>>> >>>> I just looked at the Provider base class, though, and it's not at all >>>> clear from this. Part of the problem is that the system is smart enough >>>> to >>>> skip providers that don't support prefetch, and the way we denote 'doesn't >>>> support prefetch' is that it doesn't have the method. Given that, I left >>>> the method off of the base class. Because 'instances' isn't used for this >>>> kind of test, the base class ships with a stub method that just throws an >>>> error. >>>> >>>> So yeah, all that's confusing, and could really use some additional >>>> work to make it easier to understand, and to bring some consistency. >>>> >>>> On Mar 29, 2013, at 12:37 PM, fatmcgav <[email protected] <javascript:>> >>>> wrote: >>>> >>>> Luke >>>> >>>> No worries. >>>> >>>> For the 2 I've converted soo far, it's made quite a performance >>>> difference. Thankfully can pull back all the resources with one NetApp api >>>> call, so it's drop the resource time from 5+ seconds to sub 1 second. >>>> However was a bit confused initially, as different providers seem to >>>> use different models... Some with just prefetch, some with just instances, >>>> some with both... >>>> >>>> However I'm just about to start the more complex netapp_volume >>>> provider, which has 4+ properties that have different api calls on top of >>>> the base resource list call. This is by far the slowest provider out of >>>> them with an average of 20+ seconds per run. >>>> So not sure of the best way to handle as yet. >>>> >>>> Cheers >>>> Gav >>>> On 29 Mar 2013 12:11, "Luke Kanies" <[email protected] <javascript:>> >>>> wrote: >>>> >>>>> Hi Gavin, >>>>> >>>>> I'm glad to see you've sorted it out, sorry I didn't jump in before it >>>>> was all resolved. >>>>> >>>>> How has prefetching worked out for you, in terms of performance? >>>>> >>>>> On Mar 28, 2013, at 3:28 PM, Gavin Williams >>>>> <[email protected]<javascript:>> >>>>> wrote: >>>>> >>>>> Afternoon >>>>> >>>>> Managed to find my issue.. Variable name re-use :( >>>>> >>>>> Defined 'qtrees' as an empty array on line 3, and then populated it >>>>> with a whole load of device output on line 18 :( >>>>> >>>>> So have defined a 'qtree_instances' array on the outside to contain my >>>>> output, with qtrees being used on the inside to hold the NetApp filer >>>>> response... Latest code in >>>>> Github<https://github.com/fatmcgav/fatmcgav-netapp/commit/1d51b1267466176db1a3d4e6ae32d9340a06fb56> >>>>> . >>>>> >>>>> Cheers >>>>> Gavin >>>>> >>>>> On Thursday, 28 March 2013 11:44:39 UTC, Gavin Williams wrote: >>>>>> >>>>>> Morning all >>>>>> >>>>>> Quick update... Looks like I managed to hack around the issue by >>>>>> adding the following: >>>>>> >>>>>> ... >>>>>> >>>>>> ap qtree_info >>>>>> >>>>>> # Check if it is a NaElement >>>>>> >>>>>> >>>>>> next unless qtree_info.respond_to?(:child_****get_string) >>>>>> >>>>>> # Pull out the qtree name. >>>>>> >>>>>> >>>>>> name = qtree_info.child_get_string("**q**tree") >>>>>> ... >>>>>> >>>>>> >>>>>> However this shows that I'm getting a total of 78 items processed, >>>>>> whereas the original array only contains 53 items... >>>>>> The additional items being processed are all like: '* >>>>>> #<Puppet::Type::Netapp_qtree::ProviderNetapp_qtree:*'. >>>>>> Have updated the gist with latest code and log file. >>>>>> >>>>>> Would like to understand where these are coming from, and if it's >>>>>> something I'm doing incorrectly? >>>>>> >>>>>> In the mean-time, following fixing that bug, the provider now seems >>>>>> to work as expected :) >>>>>> >>>>>> Cheers >>>>>> Gavin >>>>>> >>>>>> >>>>>> On Wednesday, 27 March 2013 17:32:23 UTC, Gavin Williams wrote: >>>>>>> >>>>>>> Afternoon all >>>>>>> >>>>>>> I've started working on converting a couple more of my NetApp >>>>>>> network device providers to use a prefetch/flush model, as can see >>>>>>> performance gains available, etc... >>>>>>> >>>>>>> Anyways, I'm having issues with my netapp_qtree provider. It would >>>>>>> appear that somehow, an additional *Puppet::Type...* row is getting >>>>>>> into an array and breaking things... >>>>>>> >>>>>>> Have created a gist here <https://gist.github.com/fatmcgav/5256240>with >>>>>>> the details, as the log file is quite long. >>>>>>> Also includes the *instances *and *prefetch* def's for my >>>>>>> netapp_qtree provider... >>>>>>> >>>>>>> As you can see on Line 337 of the log, the array contains 40 items, >>>>>>> however on line 734 *self.instances* is trying to process item >>>>>>> 41?!?! >>>>>>> What's also strange is that the item contents look like a Puppet >>>>>>> Type (*#<Puppet::Type::Netapp_qtree*)****, whereas all the others >>>>>>> in the array are NetApp specific items (*#<NaElement:*). >>>>>>> >>>>>>> So, any ideas??? >>>>>>> >>>>>>> Cheers >>>>>>> Gavin >>>>>>> >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Puppet Developers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to puppet-dev+...@**googlegroups.com <javascript:>. >>>>> To post to this group, send email to >>>>> [email protected]<javascript:> >>>>> . >>>>> Visit this group at >>>>> http://groups.google.com/**group/puppet-dev?hl=en<http://groups.google.com/group/puppet-dev?hl=en> >>>>> . >>>>> For more options, visit >>>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>>> . >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | >>>>> +1-615-594-8199 >>>>> Join us at PuppetConf 2013, August 22-23 in San Francisco - >>>>> http://bit.ly/pupconf13 >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Puppet Developers" group. >>>>> To unsubscribe from this topic, visit https://groups.google.com/d/** >>>>> topic/puppet-dev/74JW491YSAk/**unsubscribe?hl=en<https://groups.google.com/d/topic/puppet-dev/74JW491YSAk/unsubscribe?hl=en> >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> puppet-dev+...@**googlegroups.com <javascript:>. >>>>> To post to this group, send email to >>>>> [email protected]<javascript:> >>>>> . >>>>> Visit this group at >>>>> http://groups.google.com/**group/puppet-dev?hl=en<http://groups.google.com/group/puppet-dev?hl=en> >>>>> . >>>>> For more options, visit >>>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>>> . >>>>> >>>>> >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Puppet Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to puppet-dev+...@**googlegroups.com <javascript:>. >>>> To post to this group, send email to [email protected]<javascript:> >>>> . >>>> Visit this group at >>>> http://groups.google.com/**group/puppet-dev?hl=en<http://groups.google.com/group/puppet-dev?hl=en> >>>> . >>>> For more options, visit >>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>> . >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | >>>> +1-615-594-8199 >>>> Join us at PuppetConf 2013, August 22-23 in San Francisco - >>>> http://bit.ly/pupconf13 >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Puppet Developers" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/** >>>> topic/puppet-dev/74JW491YSAk/**unsubscribe?hl=en<https://groups.google.com/d/topic/puppet-dev/74JW491YSAk/unsubscribe?hl=en> >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> puppet-dev+...@**googlegroups.com <javascript:>. >>>> To post to this group, send email to [email protected]<javascript:> >>>> . >>>> Visit this group at >>>> http://groups.google.com/**group/puppet-dev?hl=en<http://groups.google.com/group/puppet-dev?hl=en> >>>> . >>>> For more options, visit >>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>> . >>>> >>>> >>>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected]<javascript:> >> . >> Visit this group at http://groups.google.com/group/puppet-dev?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> >> >> >> -- >> Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | >> +1-615-594-8199 >> Join us at PuppetConf 2013, August 22-23 in San Francisco - >> http://bit.ly/pupconf13 >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Puppet Developers" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/puppet-dev/74JW491YSAk/unsubscribe?hl=en >> . >> To unsubscribe from this group and all its topics, send an email to >> [email protected] <javascript:>. >> To post to this group, send email to [email protected]<javascript:> >> . >> Visit this group at http://groups.google.com/group/puppet-dev?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected]<javascript:> > . > Visit this group at http://groups.google.com/group/puppet-dev?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > -- > Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | > +1-615-594-8199 > Join us at PuppetConf 2013, August 22-23 in San Francisco - > http://bit.ly/pupconf13 > > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-dev?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
