Very interesting, thanks for sharing this info. btw - First time I hear about the idea to run load tests on Production during off-hours.
Shmuel Krakower. www.Beatsoo.org - re-use your jmeter scripts for application performance monitoring from worldwide locations for free. On Wed, Feb 6, 2013 at 10:21 PM, David Luu <[email protected]> wrote: > Hi Shmuel, > > No, I haven't tried the approach I mentioned yet, though I'd like to > sometime, never got the opportunity to do so. Rather I do it with a similar > less optimal approach: > > * run JMeter load test w/ X threads for some arbitrary duration (determined > by me and my colleagues) > * at same time, perform manual browser exploratory testing while load is > running to gauge UI responsively and functionality. Our goal during the > browser testing is just to gauge behavior and not functional testing. > > Since the load testing I've been doing has been roughly hands on, not > something fully automated and left to run overnight, etc. (we run our tests > on live production environment in the wee hours rather than in test > environments), there was no real need to have a Selenium test component > when it was more optimal for us to assess client side performance in simple > manual tests while watching the server load w/ our IT/Ops team. > > I would also like to add that my suggested approach applies across > different application/platform types. Web apps tend to have the luxury of > open source tools like Selenium and JMeter to assist with performance > testing, whether its super easy to use or not. You get used to it if you > come from other domains as well... > > I had worked before in telecom domain with a product that had mixed apps > (phones, desktop GUI, web, etc.). We generated load with variety of tools > (custom & commercial), and I built a simple client side performance > script/tool using free desktop GUI automation tool (commercial was too > expensive) to do some simple UI activity while the load was running and > timed the commands (start/end of command) and dumped out the timings for > analysis. As we needed to repeat this periodically, our GUI app was slow > (even w/o load), and we ran extended load overnight, this script I built > was helpful to assess client side performance than to do it manually or pay > $$$ for some commercial tool (if exist) that "supposedly" makes it easy for > a non-programmer to do same thing. Note that in my case the load was > against server, not the GUI client, but we also wanted to monitor the GUI > client responsiveness when server was under load. > > Regards, > David > > On Tue, Feb 5, 2013 at 11:16 PM, Shmuel Krakower <[email protected]> > wrote: > > > Hi David, > > I saw some solutions suggesting Selenium for load testing (so they > promise > > "easier" scripting) but not sure how accurate the numbers are and in > > general, not sure how much a single selenium load test can scale to > > internet kind of load (thousands of parallel threads/browsers)... > > > > That's where I noticed that some load testing SaaS providers (cannot > recall > > who right now), provide what you've described above. > > I like that approach, monitoring with Selenium during a JMeter load test. > > The only downside I see here, is the extra work for managing both set of > > scripts and maintaining extra infrastructure for Selenium. > > Have you ever done this? How did it work out for you? > > Did someone ever tried to use the JMeter WebDriver plugin ( > > https://github.com/cplim/JMeter), so it sounds easier to control this > > approach from a single test plan? > > > > Shmuel Krakower. > > www.Beatsoo.org - re-use your jmeter scripts for application performance > > monitoring from worldwide locations for free. > > > > > > On Wed, Feb 6, 2013 at 6:17 AM, David Luu <[email protected]> wrote: > > > > > Also, if the OP's intent is to want to test both server & client > > > performance, then in terms of JMeter, the optimal approach is to run 2 > > > tests in parallel: > > > > > > * a JMeter load test generating HTTP requests emulating the AJAX calls, > > set > > > to run for X user load (X threads) > > > * a Selenium test customized to log response times of every action > (e.g. > > > AJAX javascript click & expecting some UI response), going through same > > UI > > > flow as JMeter test but exercising real UI, this test is configured to > > run > > > in a loop continuously (might need test framework to support this or > > custom > > > code). > > > > > > The JMeter test will stress & analyze server while the Selenium test > will > > > assess the browser performance at the same time. > > > > > > That's the best approach for any toolset used for testing both clien & > > > server performance at same time. Some commercial tools just might > happen > > to > > > bundle both types of testing together. > > > > > > On Tue, Feb 5, 2013 at 6:59 PM, Stott, Charlie <[email protected]> > wrote: > > > > > > > You only need to include the javascript performance if you are > testing > > > > browser performance. > > > > > > > > I assume you are testing server performance, which is responding to > > AJAX > > > > requests, not to javascript interpretation. The interpretation > > produces > > > a > > > > request which jmeter can emulate. > > > > > > > > The clicking of the javascript button is incidental to the > performance > > of > > > > the server. If the click MUST be emulated for whatever reason we > don't > > > > understand, webdriver can do it. But there is no point testing the > > > > performance of webdriver samples this way. It is more useful if it > > > > considered part of setup and teardown. Why would you want the > > > performance > > > > of the javascript, the rendering of a button and a clicking event. > > What > > > do > > > > you gain from this? > > > > > > > > Charlie > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Deepak Shetty [mailto:[email protected]] > > > > > Sent: Wednesday, 6 February 2013 12:51 PM > > > > > To: JMeter Users List > > > > > Subject: Re: complex javascript actions in jmeter load test > > > > > > > > > > >Actually that does matter it cannot do JavaScript. If a request > > > > > >requires you need to be able to click a JavaScript button then the > > > > > >request will never happen. > > > > > The point is that what happens when the button is clicked? Assuming > > > its a > > > > > server - ajax call then A HTTP call is made and some parameters are > > > > passed > > > > > and some values are returned. Thats whats important for the load > > test , > > > > not > > > > > the fact that javascript was executed. > > > > > So when you record the script , you will be the person clicking the > > > > > button(you are recording your actions) , JMeter will record every > > > > interaction > > > > > that makes a call to the server and will record this as a separate > > HTTP > > > > > request and when you run the script the same request will be made > as > > if > > > > > someone clicked the button! > > > > > > > > > > You dont need to use the recorder either , you can modify the > script > > > > > yourself. > > > > > > > > > > If the javascript didnt actually make any server side call - then > it > > > > doesnt > > > > > matter because you dont want to load test this anyway. > > > > > > > > > > Have you actually tried this? It sounds as if you have a problem > > > > recording > > > > > your script and you probably have concluded that JMeter doesnt do > > > > > javascript (true) and hence cant test websites that do > > javascript/ajax > > > > > (false) > > > > > > > > > > >Real browser is needed > > > > > Not for a good deal of use cases - as many of the people on this > > > mailing > > > > list > > > > > can attest too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 5, 2013 at 6:44 PM, Zippy Zeppoli > > > > > <[email protected]>wrote: > > > > > > > > > > > Deepak, > > > > > > Actually that does matter it cannot do JavaScript. If a request > > > > > > requires you need to be able to click a JavaScript button then > the > > > > > > request will never happen. No request will ever be made. Also > > > testing > > > > > > true web performance requires rendering the DOM, not just > > initiating > > > > > > HTTP requests and recording the response time, rps, etc. > > > > > > > > > > > > Real browser is needed, with JavaScript, and Jmeter doesn't > > integrate > > > > > > well with this, it isn't designed for this, which is > > understandable. > > > > > > The problem is there is a gap between real browser testing (owned > > by > > > > > > third party > > > > > > companies) and open source tools (Jmeter). There's nothing in > > between > > > > > > for real-browser based performance testing. I could go into why, > > but > > > > > > its off topic of this list, and I'd rather spare everyone the > gas. > > > > > > > > > > > > Point being, Jmeter cannot solve my problem, without some serious > > > > > > customization. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 5, 2013 at 6:33 PM, Deepak Shetty <[email protected] > > > > > > > wrote: > > > > > > > > > > > > > Hi > > > > > > > You are getting too caught up in the JMeter doesnt do > javascript > > > > thing. > > > > > > In > > > > > > > most cases it doesnt matter. > > > > > > > You have a webserver that is receiving HTTP requests - whether > > > those > > > > > > > requests are generated via the user clicking a link or via AJAX > > or > > > > > > > via flash is hardly relevant to the webserver. It sees HTTP > > > requests > > > > > > > and > > > > > > sends > > > > > > > HTTP responses. > > > > > > > JMeter deals with HTTP request and responses. As long as you > can > > > > > > > make the same request that your javascript is making (which you > > can > > > > > > > see via the recording feature) , you can test it with Jmeter. > The > > > > > > > fact that the proxy cant record javascript is irrelevant - it > > only > > > > > > > needs to record the requests. If your javascript is responsible > > for > > > > > > > generating the requests > > > > > > in > > > > > > > some way , then you have to replicate that within JMeter. > > > > > > > > > > > > > > Some caveats here are > > > > > > > a. Your toolkit should be good when you need to > > parameterise(JQuery > > > > > > > is great , GWT sucks) b. If you need to perform some AJAX > > requests > > > > > > > in parallel then JMeter isnt quite there yet. > > > > > > > > > > > > > > Its probably true that it is easier writing scripts for > Selenium > > > > > > > like > > > > > > tools > > > > > > > - but they also need more resources as well as given how often > a > > UI > > > > > > changes > > > > > > > as opposed to the HTTP request/response pair , sometimes JMeter > > > > > > > scripts > > > > > > are > > > > > > > more resilient to change. > > > > > > > > > > > > > > regards > > > > > > > deepak > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 5, 2013 at 5:29 PM, Zippy Zeppoli > > > > > > > <[email protected] > > > > > > > >wrote: > > > > > > > > > > > > > > > Proxy won't work for clicking on JavaScript. > > > > > > > > Selenium will, however, Selenium isn't designed for > performance > > > > > > testing, > > > > > > > > with the exception of BrowserMob, who has perfected it. > > > > > > > > > > > > > > > > If JMeter had some kind of (good) Selenium integration then > it > > > > > > > > might be able to achieve it. > > > > > > > > > > > > > > > > I think JMeter is a good tool, but it's not built for this > kind > > > of > > > > > > work. > > > > > > > > > > > > > > > > On Tue, Feb 5, 2013 at 5:20 PM, David Luu < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > I'm not familiar with the full feature set of BrowserMob, > but > > > > > > > > > JMeter > > > > > > > can > > > > > > > > do > > > > > > > > > what you want, easy or hard, depending on your needs. > > > > > > > > > > > > > > > > > > If all you need is simple record & playback, JMeter proxy > is > > > > > > > > > best > > > > > > > option. > > > > > > > > > Just record w/ proxy, it generates a basic test plan that > you > > > > > > > > > can > > > > > > save > > > > > > > to > > > > > > > > > file and run w/ X threads to generate load, etc. > > > > > > > > > > > > > > > > > > But that approach will always use the same user, same data > in > > > > > > > > > the > > > > > > AJAX > > > > > > > > > calls during load generation. > > > > > > > > > > > > > > > > > > If you need them to be unique & parameterized, that's where > > it > > > > > > > > > takes > > > > > > > some > > > > > > > > > work to go in and cleanup the proxy recording w/ > > paramaterized > > > > > > > > > data, response checking (if more than just checking HTTP > 200 > > > > > > > > > OKs). And > > > > > > you'll > > > > > > > > > find proxy recording generates a lot of unnecessary HTTP > > > > > > > > > requests for > > > > > > > > HTML, > > > > > > > > > CSS, image files that aren't necessary of interest as > you're > > > > > > > > > testing > > > > > > > the > > > > > > > > > AJAX calls, which is where you filter out only the requests > > of > > > > > > interest > > > > > > > > and > > > > > > > > > only use those in test plan, etc. > > > > > > > > > > > > > > > > > > I believe this kind of work is needed regardless of what > tool > > > > > > > > > you > > > > > > use, > > > > > > > > some > > > > > > > > > tools just make it easier, but there's still work to do. > Not > > > > > > > > > just a > > > > > > one > > > > > > > > > touch button approach unless all you need is simple record > & > > > > > > playback. > > > > > > > > > > > > > > > > > > On Tue, Feb 5, 2013 at 5:08 PM, Zippy Zeppoli < > > > > > > [email protected] > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > > > Will probably just buy BrowserMob. > > > > > > > > > > Too bad there isn't an open source framework to already > do > > > > this. > > > > > > > > > > > > > > > > > > > > Building this is a yak shave, and I need to be testing, > not > > > > > > building > > > > > > > a > > > > > > > > > test > > > > > > > > > > harness thats probably going to break on the next > release. > > > > > > > > > > > > > > > > > > > > On Tue, Feb 5, 2013 at 3:40 PM, David Luu < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > JMeter proxy is the most integrated approach, but I > find > > it > > > > > > > > > > > gives > > > > > > > > "too > > > > > > > > > > much > > > > > > > > > > > information", so I tend to use external tools like > > browser > > > > > > traffic > > > > > > > > > > sniffers > > > > > > > > > > > (HttpFox, livehttpheaders, ieHttpHeaders) to see what > > HTTP > > > > > > requests > > > > > > > > are > > > > > > > > > > > made for AJAX calls for just the requests & responses > I'm > > > > > > > interested > > > > > > > > > in, > > > > > > > > > > > easier to manage. But that's just my opinion, others > > might > > > > > > > > > > > find > > > > > > the > > > > > > > > > > JMeter > > > > > > > > > > > proxy easier to use. > > > > > > > > > > > > > > > > > > > > > > Whatever approach you take, you just need to know what > > HTTP > > > > > > > requests > > > > > > > > > are > > > > > > > > > > > made by the AJAX calls to replicate in JMeter, and > > > > > > > > > > > parameterize > > > > > > > those > > > > > > > > > > > requests to take in dynamic/test data as needed, assert > > > > > > appropriate > > > > > > > > > > > response data, etc. In this case, WebDriver can be > > dropped > > > > > > > > > > > from > > > > > > the > > > > > > > > > > > equation, just gives more overhead and lowers > scalability > > > in > > > > > > > > > > > load generation. > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 5, 2013 at 1:55 PM, Philippe Mouawad < > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > AjaxCall will emit a network call which JMeter will > > > > capture. > > > > > > > > > > > > > > > > > > > > > > > > Use JMeter Server Proxy and you should have the Ajax > > > Calls > > > > > > > > recorded. > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > Philippe M. > > > > > > > > > > > > > > > > > > > > > > > > Follow me on twitter <https://twitter.com/philmdot> > > > > > > > > > > > > > > > > > > > > > > > > UBIK-INGENIERIE on TWITTER > > > > > > > > > > > > <https://twitter.com/ubikingenierie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > UBIK LOAD PACK BLOG < > > > http://www.ubik-ingenierie.com/blog/> > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 5, 2013 at 10:52 PM, Zippy Zeppoli < > > > > > > > > > [email protected] > > > > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > > > > > > > > > To clarify: > > > > > > > > > > > > > An example would be: > > > > > > > > > > > > > 1) log in via a form post > > > > > > > > > > > > > 2) look at orders in an ecommerce interface (AJAX > > call) > > > > > > > > > > > > > 3) click on result to view order detail (AJAX) > > > > > > > > > > > > > > > > > > > > > > > > > > Pretty much a standard ecommerce transaction, but > the > > > > > > interface > > > > > > > > is > > > > > > > > > > all > > > > > > > > > > > > > javascript. > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 4, 2013 at 7:40 PM, Stott, Charlie < > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > > > > > From: David Luu [mailto:[email protected]] > > > > > > > > > > > > > > > Sent: Tuesday, 5 February 2013 1:18 PM > > > > > > > > > > > > > > > To: JMeter Users List > > > > > > > > > > > > > > > Subject: Re: complex javascript actions in > jmeter > > > > > > > > > > > > > > > load > > > > > > test > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> You can use webdriver from jmeter. Create > a > > > > > > webdriver > > > > > > > > > class > > > > > > > > > > > that > > > > > > > > > > > > > > > performs the requests and runs the javascript > via > > > > > > > > > > > > > > > the > > > > > > > > browser, > > > > > > > > > > then > > > > > > > > > > > > > > run/call > > > > > > > > > > > > > > > it from a BSF or JSR sampler. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just to clarify, I take it that's only > worthwhile > > > to > > > > > > > > > > > > > > > do > > > > > > (in > > > > > > > > > terms > > > > > > > > > > > of > > > > > > > > > > > > > > > scalability) when using PhantomJSDriver or > > > > > > > > > > > > > > > HtmlUnitDriver > > > > > > > or > > > > > > > > > > > > > > FirefoxDriver > > > > > > > > > > > > > > > (on Linux with xvfb) with JMeter this way? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "Worthwhile" depends on assumptions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Because otherwise, the browser GUI is the > > > > > > > > > > > > > > > scalability > > > > > > > > limiting > > > > > > > > > > > factor > > > > > > > > > > > > > > even > > > > > > > > > > > > > > > with JMeter and Grid deployment, and in that > > case, > > > > > > > > > > > > > > > no > > > > > > > > > difference > > > > > > > > > > in > > > > > > > > > > > > > using > > > > > > > > > > > > > > > WebDriver outside JMeter to do performance > tests > > > > > > > > > > > > > > > except > > > > > > if > > > > > > > > one > > > > > > > > > > > wants > > > > > > > > > > > > > the > > > > > > > > > > > > > > > JMeter logging/reporting facilities to help > > > > > > > > > > > > > > > performance > > > > > > > test, > > > > > > > > > > > because > > > > > > > > > > > > > > there's > > > > > > > > > > > > > > > no or minimal scalabiity gain. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Seems the poster is more at the > > capability/viability > > > > > > > > > > > > > > stage > > > > > > of > > > > > > > > > > > > developing > > > > > > > > > > > > > > tests? We would need much more information to > > start > > > > > > advising > > > > > > > > on > > > > > > > > > > the > > > > > > > > > > > > > entire > > > > > > > > > > > > > > load testing process, start to finish, and what > > > > > > > > > > > > > > challenges > > > > > > > may > > > > > > > > > face > > > > > > > > > > > the > > > > > > > > > > > > > > poster along the way. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 4, 2013 at 4:42 PM, Stott, Charlie > < > > > > > > > > > [email protected]> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You can use webdriver from jmeter. Create a > > > > > > > > > > > > > > > > webdriver > > > > > > > > class > > > > > > > > > > that > > > > > > > > > > > > > > > > performs the requests and runs the javascript > > via > > > > > > > > > > > > > > > > the > > > > > > > > > browser, > > > > > > > > > > > then > > > > > > > > > > > > > > > > run/call it from a BSF or JSR sampler. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > > > > > > > From: Zippy Zeppoli > > > > > > > > > > > > > > > > > [mailto:[email protected]] > > > > > > > > > > > > > > > > > Sent: Tuesday, 5 February 2013 9:28 AM > > > > > > > > > > > > > > > > > To: JMeter Users List > > > > > > > > > > > > > > > > > Subject: Re: complex javascript actions in > > > > > > > > > > > > > > > > > jmeter > > > > > > load > > > > > > > > test > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The problem is Selenium has no performance > > > > > > > > > > > > > > > > > testing > > > > > > > > harness. > > > > > > > > > > > > > > > > > Sucks that it seems BrowserMob (paid > > solution) > > > > > > > > > > > > > > > > > is the > > > > > > > > only > > > > > > > > > > > solid > > > > > > > > > > > > > > option. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Until someone builds something with > > Phantom.js, > > > > > > > > > > > > > > > > > but > > > > > > it > > > > > > > > > seems > > > > > > > > > > > > JMeter > > > > > > > > > > > > > > > > > isn't going to cut it here. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Feb 1, 2013 at 5:40 PM, David Luu < > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You'll need to figure out what the > complex > > > > > > javascript > > > > > > > > > does. > > > > > > > > > > > > Does > > > > > > > > > > > > > > > > > > it make any AJAX requests, or is it all > > local > > > > > > client > > > > > > > > side > > > > > > > > > > > > > > > > processing/rendering? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If it's all local, then there's no point > > > > > > > > > > > > > > > > > > testing it > > > > > > > > with > > > > > > > > > > > > JMeter, > > > > > > > > > > > > > > > > > > that's client side browser testing better > > > done > > > > > > > > > > > > > > > > > > with > > > > > > > > > > Selenium. > > > > > > > > > > > > It > > > > > > > > > > > > > > > > > > won't impact the server side load test > > > (except > > > > > > delay > > > > > > > in > > > > > > > > > > > server > > > > > > > > > > > > > > > > > > response time for fetching files will > > impact > > > > > > > > > > > > > > > > > > the > > > > > > > > > javascript > > > > > > > > > > > > > > > > > > execution on client side, but that can be > > > > > > compensated > > > > > > > > w/ > > > > > > > > > > > JMeter > > > > > > > > > > > > > > > > > > load test against server with 1+ Selenium > > > test > > > > > > > running > > > > > > > > at > > > > > > > > > > > same > > > > > > > > > > > > > > > > > > time to gauge client side performance of > > > > > > > > > > > > > > > > > > site/app > > > > > > in > > > > > > > > > > > browser). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If the javascript does execute AJAX > > requests, > > > > > > > > > > > > > > > > > > you > > > > > > > need > > > > > > > > to > > > > > > > > > > > > figure > > > > > > > > > > > > > > > > > > out the HTTP requests made and mimic that > > in > > > > > > > > > > > > > > > > > > JMeter > > > > > > > as > > > > > > > > > part > > > > > > > > > > > of > > > > > > > > > > > > > your > > > > > > > > > > > > > > > test. > > > > > > > > > > > > > > > > > > You can get that reading dev/design docs, > > or > > > > > > reverse > > > > > > > > > > > > > > > > > > engineer/traffic sniffing the app while > > doing > > > > > > manual > > > > > > > > > > testing. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Feb 1, 2013 at 1:45 PM, Zippy > > Zeppoli > > > > > > > > > > > > > > > > > > <[email protected] > > > > > > > > > > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > If I have a website which requires > > logging > > > > > > > > > > > > > > > > > > > in, > > > > > > and > > > > > > > > > > > executing > > > > > > > > > > > > > > > > > > > complex javascript actions, how would I > > do > > > > > > > > > > > > > > > > > > > this > > > > > > (if > > > > > > > > at > > > > > > > > > > all) > > > > > > > > > > > > in > > > > > > > > > > > > > > jmeter? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've heard of writing groovy scripts to > > do > > > > > > > > > > > > > > > > > > > this > > > > > > but > > > > > > > > > this > > > > > > > > > > > > sounds > > > > > > > > > > > > > > > > > > > like a > > > > > > > > > > > > > > > > > > lot > > > > > > > > > > > > > > > > > > > of work / maintenance. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------ > > > > > > > > --- > > > > > > > > > > > > > > > > To unsubscribe, e-mail: > > > > > > > [email protected] > > > > > > > > > > > > > > > > For additional commands, e-mail: > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > > > To unsubscribe, e-mail: > > > > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > For additional commands, e-mail: > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > >
