Hello, I used to have the same issue as you, only that I had to collect data from several machines, each file having 8-15Gb after 1-2 days of testing (i.e. a test over weekend).
Possible approaches: 1. gnuplot: works better with large sets of data and you can start the command line when jmeter stops doing the load. because it doesn't plot when it analyzes the data, it finishes fast and gives you a .png. Downside: scripts are pretty dependent on format of data from that particular test and the learning curve is high. 2. either send statistical data to a sql DB instance (during the test) or import csv into a DB after the test. there are tools that help with ploting from a mysql. I've chosen no 2, but immediately after that I ran into problems with the size of the DB because of the high number of new entries (we're talking about tens of millions of lines). The advantage of a DB is that when you take test results from several machines, it helps with ordering by timestamp, so you can plot what happened on all machines easier. The problem with the size of the DB got fixed with a workaround, using more tables (per day, per week, whatever), but I can't claim that fix because I got help from both devs and the IT dev ops. 3. use application level monitoring tools. You don't even need jmeter results unless you doubt the accuracy of the tool. Production apps need these tools anyway, so set them early on. ------ So here is where I wanted to reach and get your input on this as well: Why do you even need that much data? For granularity of the graph? Then why not run shorter tests? My approach, today, to this issue is that 10% of the samples from a large set of data is just as relevant as the whole set (sometimes you do need to run longer tests with an extreme number of requests). So, I either save only 1 in 10 samples that I'm particullary interested in (maybe less sometimes), or save on disk from only 1 jmeter instance out of ten. I just make sure that whatever approach I choose doesn't affect performance client side, but usually - less logging means better performance. If I run endurance tests from 3-5 server class test machines, then I also use my desktop to generate <1% of the total load, to get live feedback about what is going on. The script on my desktop looks like a heartbeat in comparison to the load generators (I just take the pulse of the application). This makes post analysis of the other files pretty much redundant and it doesn't bother me to restart the monitoring instance of jmeter from my desktop. Reasoning: statistically, if the data set is relevant and has enough repetitions then 10k, 100k samples are just as relevant as 10 million. So why bother with working with the large set in the first place? All I need to see is how response times fluctuated throughout the test period, this can be done with the methods I've used. What do you think? I hope you or somebody can give me some examples when this is not a good practice. Its not the first time people ask questions about this, I hope more contribute with their experience on this thread, because a lot of people can benefit from hands-on-approaches to this problem. Thank you, Adrian S On Wed, Jun 20, 2012 at 1:36 PM, Shay Ginsbourg <sginsbo...@gmail.com>wrote: > Hi, > > Thanks for the reply. > > Q: Is there a recommended configuration setting for opening large JTL files > within Jmeter ? > > For example, while running a larger JTL (>> 10GB), a see that the java.exe > CPU is < 10% on Win7 task manager. > > It takes much time to create just the Jmeter summary report on a i5 core > with 6 GB RAM. > > Probably, the bottleneck is the speed of the hard-drive, but I'm wandering > if anything could be optimized at some level. > > thanks, > Shay > > > > > On Mon, Jun 18, 2012 at 1:54 AM, oliver lloyd <oliver_ll...@hotmail.com > >wrote: > > > There are options for minimising the amount of data stored, such as: > > > > Use csv format > > Only log the data you actually need by setting options in the properties > > (these are well documented in the file itself). > > > > But this will only improve things up to a point, ultimately if your tests > > are making large numbers of requests then you will have large numbers of > > results. The solution is to aggregate the data after the test run is > > complete. > > > > One quick option to do this is to write a simple awk script; you could > > even get a bit clever and pass in the total row count of the file which > > would allow the aggregation to be dynamic (ie. instead of grouping by > every > > 15 lines it would group by every n lines where n is a calculated value > > based on how many rows there are). The nice thing about using something > > like awk is you can very easily wrap the whole process in a shell script > > (running jmeter and processing the results) and then everything is > > automated. Perl is also nice for this sort of thing but my preferred > > solution is to import the data to a database and use queries to reduce > the > > output as this comes with a number of rather useful additional benefits > > (trending of results, comparisons, etc.) but then this option is a little > > more work to setup. > > > > > > On 17 Jun 2012, at 23:27, Shay Ginsbourg wrote: > > > > > Hi, > > > > > > What should be done in order to reduce the file size of the JTL results > > > while running Jmeter CMD ? > > > > > > The problem is that after a meaningful long run, we get over 20 GB, > which > > > is way too much of a file. > > > > > > regards, > > > Shay > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: user-unsubscr...@jmeter.apache.org > > For additional commands, e-mail: user-h...@jmeter.apache.org > > > > > > > -- > > > Shay Ginsbourg > > Regulatory & Testing Affairs Consultant > > > WWW.GINSBOURG.COM > > > Providing Regulatory, Medical & Performance Testing services since 2008: > > * IEC 62304 Medical Device Software Life Cycle > > * IEEE 829 Software Test Documentation > > * ISO 14971 Medical Device Risk Management > > * FDA 21 CFR Part 11 Software Validation > > * IEC 60601-1:2005 3rd ED PEMS - Medical Electrical Equipment > > * End-to-end verification, validation, and testing (VV&T) > > * FDA and CE submissions > > * Open source free testing tools implementation > > * Functionality and regression testing > > * Software Performance & Load testing > > * Software Testing Advanced Automation > > * Medical Software Verification & Validation > > * Medical Device Verification & Validation > > * Medical Device Regulatory Submission > > * Organizational Regulatory Qualification > > > Formerly QA Manager of LoadRunner at Mercury Interactive > > > M.Sc. cum laude in Bio-Medical Engineering > > M.Sc. in Mechanical Engineering > > > > Work: +972(0)3-5185873 > > Mobile: +972(0)54-6690915 > > > Email: sginsbo...@gmail.com > > > Visit my personal page on LinkedIn at: > http://www.linkedin.com/in/shayginsbourg > > > Please consider your environmental responsibility before printing this > e-mail. >
