Excellent. On Apr 5, 2013, at 12:33 PM, fatmcgav <[email protected]> wrote:
> I was looking at doing all the work in the 'flush' method, however have just > been doing some more testing and think I've come up with an alternative... > > Can run the user creation in the 'create' def, and then clear @property_hash > so that it doesn't get invoked in 'flush'. > 'Flush' then calls the user modification if required... > > Seems to work quite well from my testing :) > > Latest code is: > https://github.com/fatmcgav/fatmcgav-netapp/blob/master/lib/puppet/provider/netapp_user/netapp_user.rb > > Cheers > Gavin > > > On 5 April 2013 11:15, Luke Kanies <[email protected]> wrote: > Ah. You should copy the model in the user/group providers: The providers > have a 'create' method that uses useradd, for instance, and then a 'uid=' > method that uses usermod. > > Does that help? Are you doing all of the actual work in the 'flush' method, > and that's where you need to differentiate? > > On Apr 5, 2013, at 11:06 AM, fatmcgav <[email protected]> wrote: > >> Luke >> >> A user create requires a different API call to a modify... >> (useradmin-user-add vs useradmin-user-modify). >> >> So I could leave the creation to 'create', however how do I then trigger a >> modify in flush when required? >> Or should I treat any call into flush with a :present ensure as a modify? >> Though I vaguely remember that causing a double-call on a resource create - >> Once from create, and then another from flush... >> >> Cheers >> Gavin >> >> >> On 5 April 2013 10:01, Luke Kanies <[email protected]> wrote: >> Can you explain why you need to know the difference? >> >> The framework should handle that for you. >> >> On Apr 4, 2013, at 2:48 PM, Gavin Williams <[email protected]> wrote: >> >>> Indeed... :) >>> >>> Ok, next question for you/all... >>> >>> How could I differentiate between what is a resource creation and a >>> resource modification? >>> >>> Have updated the netapp_user provider to prefetch/flush, however am >>> struggling to differentiate between what is a resource creation, and what >>> is a modification... >>> Code is here: >>> https://github.com/fatmcgav/fatmcgav-netapp/commit/66092978f4182c5474a60011db99ee2e3e12e689 >>> >>> Thoughts? >>> >>> Cheers >>> Gavin >>> >>> On Tuesday, 2 April 2013 16:17:33 UTC+1, Luke Kanies wrote: >>> That is awesome. It's amazing what a bit of optimization will do for you. >>> >>> On Apr 2, 2013, at 3:12 PM, fatmcgav <[email protected]> wrote: >>> >>>> Yep, have been testing it today on my dev and production Puppet masters, >>>> and seems to be working as expected... >>>> >>>> Have been very impressed with the performance improvements - Has dropped >>>> the netapp_volume resource time from 40+ seconds to <0.1 seconds. So >>>> MASSIVE improvements to be made from switching to the prefetch/flush >>>> style... >>>> >>>> Cheers >>>> Gav >>>> >>>> >>>> On 2 April 2013 13:24, Luke Kanies <[email protected]> wrote: >>>> Looks good from a cursory glance. Hopefully it's all working for you. >>>> >>>> On Apr 2, 2013, at 9:52 AM, Gavin Williams <[email protected]> wrote: >>>> >>>>> As a quick update, I've pushed my latest set of code tweaks up - >>>>> https://github.com/fatmcgav/fatmcgav-netapp/commit/7a7d18bf39cdbb04a3b0b5192929ec6a857c0a5e >>>>> >>>>> Need to tidy the code up a bit, but should be able to get the gist :) >>>>> Have got a couple of defs which pull back the bulk of the information, >>>>> and then a couple that need to be called on a volume by volume basis. >>>>> All gets put together in instances, and provider'd up in prefetch. >>>>> >>>>> Comments welcome. >>>>> >>>>> Cheers >>>>> Gavin >>>>> >>>>> On Friday, 29 March 2013 13:15:12 UTC, Gavin Williams wrote: >>>>> Luke >>>>> >>>>> Cheers for that info, should prive useful... >>>>> >>>>> Guess I could implement a method for each property that pulls back all >>>>> that property values for all the volumes, and then match them up in >>>>> prefetch. >>>>> >>>>> Cheers >>>>> Gav >>>>> >>>>> On 29 Mar 2013 12:49, "Luke Kanies" <[email protected]> wrote: >>>>> Hi Gavin, >>>>> >>>>> Yeah, there isn't as much consistency in API as I would like. >>>>> >>>>> The way it should work, I believe, is something like: >>>>> >>>>> * The Type should support an 'instances' method that returns a complete >>>>> list of all instances of that type that exist on the host itself. This >>>>> method should not require a catalog. >>>>> >>>>> * The provider should support a 'prefetch' method, which accepts a list >>>>> of resources from the catalog and updates them with the correct provider >>>>> instance, with appropriate data already filled in to reflect the system >>>>> state. >>>>> >>>>> * Optionally, many providers implement (I think) their own 'instances' >>>>> method, which returns all instances and doesn't require a catalog, to >>>>> implement both of the above methods. >>>>> >>>>> I just looked at the Provider base class, though, and it's not at all >>>>> clear from this. Part of the problem is that the system is smart enough >>>>> to skip providers that don't support prefetch, and the way we denote >>>>> 'doesn't support prefetch' is that it doesn't have the method. Given >>>>> that, I left the method off of the base class. Because 'instances' isn't >>>>> used for this kind of test, the base class ships with a stub method that >>>>> just throws an error. >>>>> >>>>> So yeah, all that's confusing, and could really use some additional work >>>>> to make it easier to understand, and to bring some consistency. >>>>> >>>>> On Mar 29, 2013, at 12:37 PM, fatmcgav <[email protected]> wrote: >>>>> >>>>>> Luke >>>>>> >>>>>> No worries. >>>>>> >>>>>> For the 2 I've converted soo far, it's made quite a performance >>>>>> difference. Thankfully can pull back all the resources with one NetApp >>>>>> api call, so it's drop the resource time from 5+ seconds to sub 1 >>>>>> second. >>>>>> However was a bit confused initially, as different providers seem to use >>>>>> different models... Some with just prefetch, some with just instances, >>>>>> some with both... >>>>>> >>>>>> However I'm just about to start the more complex netapp_volume provider, >>>>>> which has 4+ properties that have different api calls on top of the base >>>>>> resource list call. This is by far the slowest provider out of them with >>>>>> an average of 20+ seconds per run. >>>>>> So not sure of the best way to handle as yet. >>>>>> >>>>>> Cheers >>>>>> Gav >>>>>> >>>>>> On 29 Mar 2013 12:11, "Luke Kanies" <[email protected]> wrote: >>>>>> Hi Gavin, >>>>>> >>>>>> I'm glad to see you've sorted it out, sorry I didn't jump in before it >>>>>> was all resolved. >>>>>> >>>>>> How has prefetching worked out for you, in terms of performance? >>>>>> >>>>>> On Mar 28, 2013, at 3:28 PM, Gavin Williams <[email protected]> wrote: >>>>>> >>>>>>> Afternoon >>>>>>> >>>>>>> Managed to find my issue.. Variable name re-use :( >>>>>>> >>>>>>> Defined 'qtrees' as an empty array on line 3, and then populated it >>>>>>> with a whole load of device output on line 18 :( >>>>>>> >>>>>>> So have defined a 'qtree_instances' array on the outside to contain my >>>>>>> output, with qtrees being used on the inside to hold the NetApp filer >>>>>>> response... Latest code in Github. >>>>>>> >>>>>>> Cheers >>>>>>> Gavin >>>>>>> >>>>>>> On Thursday, 28 March 2013 11:44:39 UTC, Gavin Williams wrote: >>>>>>> Morning all >>>>>>> >>>>>>> Quick update... Looks like I managed to hack around the issue by adding >>>>>>> the following: >>>>>>> >>>>>>> ... >>>>>>> ap qtree_info >>>>>>> >>>>>>> # Check if it is a NaElement >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> next unless qtree_info.respond_to?(:child_get_string) >>>>>>> >>>>>>> # Pull out the qtree name. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> name = qtree_info.child_get_string("qtree") >>>>>>> ... >>>>>>> >>>>>>> However this shows that I'm getting a total of 78 items processed, >>>>>>> whereas the original array only contains 53 items... >>>>>>> The additional items being processed are all like: >>>>>>> '#<Puppet::Type::Netapp_qtree::ProviderNetapp_qtree:'. >>>>>>> Have updated the gist with latest code and log file. >>>>>>> >>>>>>> Would like to understand where these are coming from, and if it's >>>>>>> something I'm doing incorrectly? >>>>>>> >>>>>>> In the mean-time, following fixing that bug, the provider now seems to >>>>>>> work as expected :) >>>>>>> >>>>>>> Cheers >>>>>>> Gavin >>>>>>> >>>>>>> >>>>>>> On Wednesday, 27 March 2013 17:32:23 UTC, Gavin Williams wrote: >>>>>>> Afternoon all >>>>>>> >>>>>>> I've started working on converting a couple more of my NetApp network >>>>>>> device providers to use a prefetch/flush model, as can see performance >>>>>>> gains available, etc... >>>>>>> >>>>>>> Anyways, I'm having issues with my netapp_qtree provider. It would >>>>>>> appear that somehow, an additional Puppet::Type... row is getting into >>>>>>> an array and breaking things... >>>>>>> >>>>>>> Have created a gist here with the details, as the log file is quite >>>>>>> long. >>>>>>> Also includes the instances and prefetch def's for my netapp_qtree >>>>>>> provider... >>>>>>> >>>>>>> As you can see on Line 337 of the log, the array contains 40 items, >>>>>>> however on line 734 self.instances is trying to process item 41?!?! >>>>>>> What's also strange is that the item contents look like a Puppet Type >>>>>>> (#<Puppet::Type::Netapp_qtree), whereas all the others in the array are >>>>>>> NetApp specific items (#<NaElement:). >>>>>>> >>>>>>> So, any ideas??? >>>>>>> >>>>>>> Cheers >>>>>>> Gavin >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Puppet Developers" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to [email protected]. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> Visit this group at http://groups.google.com/group/puppet-dev?hl=en. >>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | >>>>>> +1-615-594-8199 >>>>>> Join us at PuppetConf 2013, August 22-23 in San Francisco - >>>>>> http://bit.ly/pupconf13 >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in the >>>>>> Google Groups "Puppet Developers" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/puppet-dev/74JW491YSAk/unsubscribe?hl=en. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> Visit this group at http://groups.google.com/group/puppet-dev?hl=en. >>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Puppet Developers" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> Visit this group at http://groups.google.com/group/puppet-dev?hl=en. >>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | >>>>> +1-615-594-8199 >>>>> Join us at PuppetConf 2013, August 22-23 in San Francisco - >>>>> http://bit.ly/pupconf13 >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Puppet Developers" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/puppet-dev/74JW491YSAk/unsubscribe?hl=en. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/puppet-dev?hl=en. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Puppet Developers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/puppet-dev?hl=en. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>>> >>>> >>>> >>>> -- >>>> Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | >>>> +1-615-594-8199 >>>> Join us at PuppetConf 2013, August 22-23 in San Francisco - >>>> http://bit.ly/pupconf13 >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Puppet Developers" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/puppet-dev/74JW491YSAk/unsubscribe?hl=en. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/puppet-dev?hl=en. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Puppet Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/puppet-dev?hl=en. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> >>> >>> >>> -- >>> Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | +1-615-594-8199 >>> Join us at PuppetConf 2013, August 22-23 in San Francisco - >>> http://bit.ly/pupconf13 >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Puppet Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/puppet-dev?hl=en. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >> >> >> -- >> Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | +1-615-594-8199 >> Join us at PuppetConf 2013, August 22-23 in San Francisco - >> http://bit.ly/pupconf13 >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Puppet Developers" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/puppet-dev/74JW491YSAk/unsubscribe?hl=en. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/puppet-dev?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/puppet-dev?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > > -- > Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | +1-615-594-8199 > Join us at PuppetConf 2013, August 22-23 in San Francisco - > http://bit.ly/pupconf13 > > > -- > You received this message because you are subscribed to a topic in the Google > Groups "Puppet Developers" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/puppet-dev/74JW491YSAk/unsubscribe?hl=en. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/puppet-dev?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/puppet-dev?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > -- Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | +1-615-594-8199 Join us at PuppetConf 2013, August 22-23 in San Francisco - http://bit.ly/pupconf13 -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-dev?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
