Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-10-16 Thread Luke Kanies
On Oct 15, 2013, at 1:58 PM, Patrick Carlisle wrote: > > On Tue, Oct 15, 2013 at 1:50 PM, Nick Lewis wrote: >> If, in the course of implementing this, you happen to find it easy to >> implement an easy way to delay resources that might not be "ready", that >> would be awesome, too. That is,

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-10-15 Thread Patrick Carlisle
On Tue, Oct 15, 2013 at 1:50 PM, Nick Lewis wrote: > If, in the course of implementing this, you happen to find it easy to > implement an easy way to delay resources that might not be "ready", that > would be awesome, too. That is, if there were a hook similar to those > below that could be used

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-10-15 Thread Nick Lewis
On Tuesday, October 15, 2013 at 8:48 AM, Luke Kanies wrote: > That sounds awesome, Kylo. > > If, in the course of implementing this, you happen to find it easy to > implement an easy way to delay resources that might not be "ready", that > would be awesome, too. That is, if there were a hook si

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-10-15 Thread Kylo Ginsberg
On Tue, Oct 15, 2013 at 8:48 AM, Luke Kanies wrote: > If, in the course of implementing this, you happen to find it easy to > implement an easy way to delay resources that might not be "ready", that > would be awesome, too. That is, if there were a hook similar to those > below that could be use

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-10-15 Thread Luke Kanies
That sounds awesome, Kylo. If, in the course of implementing this, you happen to find it easy to implement an easy way to delay resources that might not be "ready", that would be awesome, too. That is, if there were a hook similar to those below that could be used to see if a resource is ready

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-10-14 Thread Kylo Ginsberg
This email thread died out but the issue has not been forgotten. Andy and I discussed this issue and came up with an initial plan for batch processing. Our thinking is to start conservatively and build the feature out from there. The new provider interface would be the one Andy suggested: >

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-09-19 Thread John Bollinger
On Wednesday, September 18, 2013 9:07:12 AM UTC-5, Andy Parker wrote: > > On Tue, Sep 17, 2013 at 1:19 PM, John Bollinger > > > wrote: > >> >> >> On Monday, September 16, 2013 11:12:58 AM UTC-5, Andy Parker wrote: >> >>> On Mon, Sep 16, 2013 at 6:56 AM, John Bollinger >>> wrote: >>>

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-09-18 Thread Andy Parker
On Tue, Sep 17, 2013 at 1:19 PM, John Bollinger wrote: > > > On Monday, September 16, 2013 11:12:58 AM UTC-5, Andy Parker wrote: > >> On Mon, Sep 16, 2013 at 6:56 AM, John Bollinger wrote: >> >>> >>> >>> On Monday, September 16, 2013 6:48:17 AM UTC-5, Andy Parker wrote: The problem

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-09-17 Thread John Bollinger
On Monday, September 16, 2013 11:12:58 AM UTC-5, Andy Parker wrote: > > On Mon, Sep 16, 2013 at 6:56 AM, John Bollinger > > > wrote: > >> >> >> On Monday, September 16, 2013 6:48:17 AM UTC-5, Andy Parker wrote: >>> >>> >>> The problem with this picture for being able to batch operations >>> to

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-09-17 Thread Andy Parker
I was able to get a bit of info about what the read queue is as the catalog is being executed. It looks like there are some times (this is just by eyeballing it, haven't crunched any numbers) where we can get a run of several packages that could be batched together. The branch that contains the ch

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-09-17 Thread Erik Dalén
There's some places where there is 19 packages on the ready queue in that example catalog. On 17 September 2013 18:48, Andy Parker wrote: > I was able to get a bit of info about what the read queue is as the > catalog is being executed. It looks like there are some times (this is just > by eyeb

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-09-16 Thread Andy Parker
On Mon, Sep 16, 2013 at 6:56 AM, John Bollinger wrote: > > > On Monday, September 16, 2013 6:48:17 AM UTC-5, Andy Parker wrote: >> >> >> The problem with this picture for being able to batch operations >> together, is that everything turns into calls on the Puppet::Type instance, >> which then mak

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-09-16 Thread John Bollinger
On Monday, September 16, 2013 6:48:17 AM UTC-5, Andy Parker wrote: > > > The problem with this picture for being able to batch operations together, > is that everything turns into calls on the Puppet::Type instance, which > then makes all of the individual calls to the provider. To batch, we ne

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-09-16 Thread Andy Parker
On Sun, Sep 15, 2013 at 11:27 PM, Erik Dalén wrote: > > > > On 13 September 2013 18:52, Henrik Lindberg < > henrik.lindb...@cloudsmith.com> wrote: > >> Hi, >> Ideas regarding a potential performance boost that can be gained by >> performing batch processing of package installs/operations has been

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-09-15 Thread Erik Dalén
On 13 September 2013 18:52, Henrik Lindberg wrote: > Hi, > Ideas regarding a potential performance boost that can be gained by > performing batch processing of package installs/operations has been > floating around in the Puppet echo system for quite some time. > > There is a discussion (and a som

Re: [Puppet-dev] Ideas for Batch Processing of Packages

2013-09-15 Thread Luke Kanies
Hi Henrik, I know we have some users who just batch all package installs up front. It'd be interesting to see if that was a feasible solution. it would by pass the graph entirely, which I'm sure could have problems, but it would, at least, be easy to build and understand. Would that suffice

[Puppet-dev] Ideas for Batch Processing of Packages

2013-09-13 Thread Henrik Lindberg
Hi, Ideas regarding a potential performance boost that can be gained by performing batch processing of package installs/operations has been floating around in the Puppet echo system for quite some time. There is a discussion (and a somewhat dated implementation/proposal) in http://projects.pu