Thanks for your info and advice. I was certain that this is not a linux problem per say, But rather a tuning issue, and now you have supplied the proof :)
Can you please send me the set of environment parameters on your machine? (Including any other relevant information such as exact JVM version, is it hotspot or not, is it a server or client impl etc.) Also, I ran the following on Linux 2.4.20-13.7 i686 java version "1.4.1_01": [EMAIL PROTECTED] run]# java -Xmx128M -Xms128M -cp .:poi-2.5.1-final-20040804.jar TestWorkbook_v251 400 10 create workBook millis=936 write workBook millis=19557 entire test millis=20493 [EMAIL PROTECTED] run]# java -Xmx512M -Xms512M -cp .:poi-2.5.1-final-20040804.jar TestWorkbook_v251 400 10 create workBook millis=937 write workBook millis=18274 entire test millis=19211 [EMAIL PROTECTED] run]# java -cp .:poi-2.5.1-final-20040804.jar TestWorkbook_v251 400 10 create workBook millis=1089 write workBook millis=20247 entire test millis=21336 Seems like memory is not my problem right? There is almost no change in performance... -----Original Message----- From: Amol Deshmukh [mailto:[EMAIL PROTECTED] Sent: Thursday, August 18, 2005 4:36 PM To: 'POI Users List' Subject: RE: bad performance in linux Zach, ** ON MY LINUX ** Here's my figures for your test on dual proc PIII Coppermine running linux2.4.18 and IBM's JRE 1.4.2: [dev001]$ java poitest.PerfTest 400 10 create workBook millis=1893 write workBook millis=942 entire test millis=2835 ** LARGE WORKBOOKS ** For larger workbooks, JVM memory allocation could significantly affect the execution time. Here's some figures to give you an idea of how in this case the Xms setting can affect execution times: -- Linux2.4.18/IBMjre1.4.2/P3 -- [dev001]$ java -Xmx128M -Xms128M poitest.PerfTest 1000 100 create workBook millis=7592 write workBook millis=2752 entire test millis=10344 [dev001]$ java -Xmx128M -Xms32M poitest.PerfTest 1000 100 create workBook millis=35778 <-- ~5x more!! write workBook millis=67625 <-- ~24x more!! entire test millis=103403 <-- 10x more!! -- Windows/SUNjre1.4.2/P4 -- > java -Xmx128M -Xms128M poitest.PerfTest 1000 100 create workBook millis=5031 write workBook millis=1922 entire test millis=6953 > java -Xmx128M -Xms32M poitest.PerfTest 1000 100 create workBook millis=5625 write workBook millis=1812 entire test millis=7437 <-- modest increase ** LINUX/Windows ** Ofcourse the real execution time would be measured in terms of CPU time, since in my observation, on Linux CPU utilization goes up to 83% max whereas on windows the peak CPU utilization goes up to 100% during execution of your test program. ** IMHO ** In general, I think you should probably look at perf tuning params in the following order: 0. JVM params (-Xmx etc.) 1. JVM upgrade (newer version/different vendor) 2. JVM params (after upgrade) After that you could look at other options like filesystem (on linux) and getting better hardware. To determine the need for change of filesystem, run tests that do plain IO (not using POI) and compare the times. HTH, ~ amol > -----Original Message----- > From: Zach Engel [mailto:[EMAIL PROTECTED] > Sent: Thursday, August 18, 2005 7:51 AM > To: 'POI Users List' > Subject: RE: bad performance in linux > > > Hi > > Sadly, I am not a unix / linux guy, so can anyone (Marx > Stefan?) refer me to > some info / best practices on the subject of Linux environment tuning? > > in the mean time I created a simple java class that creates a simple > workbook. (see code below) > > Look at the differences between windows and linux > > > performance in windows > ====================== > > Using poi version 1.5.1 > D:\JbuilderProjects\src\general\POI-SRC\run>java -cp > .;jakarta-poi-1.5.1-final-20020615.jar TestWorkbook_v151 400 10 > create workBook millis=562 > write workBook millis=125 > entire test millis=687 > > > Using poi version 2.5.1 > D:\JbuilderProjects\src\general\POI-SRC\run>java -cp > .;poi-2.5.1-final-20040804.jar TestWorkbook_v251 400 10 > create workBook millis=500 > write workBook millis=218 > entire test millis=718 > > > > > > > performance in linux > ====================== > > Using poi version 1.5.1 > [EMAIL PROTECTED] run]# java -cp > .:jakarta-poi-1.5.1-final-20020615.jar > TestWorkbook_v151 400 10 > create workBook millis=1237 > write workBook millis=275 > entire test millis=1512 > > > Using poi version 2.5.1 > [EMAIL PROTECTED] run]# java -cp .:poi-2.5.1-final-20040804.jar > TestWorkbook_v251 400 10 > create workBook millis=1099 > write workBook millis=18959 > entire test millis=20058 > > > > > > > > as you can see, in linux, > generating a sheet with 400 rows, 10 cells in each row takes 1512 > milliseconds while using a very old version of POI (1.5.1), > however, when using the newest version (2.5.1) - > it takes 20058 milliseconds! > > > I don't know if it is important, but here are the environment > variables of > both environments: > > > linux > ======================= > [EMAIL PROTECTED] run]# uname -srm > Linux 2.4.20-13.7 i686 > > [EMAIL PROTECTED] run]# env > PWD=/tmp/zach/run > VENDOR=intel > REMOTEHOST=zaksrv2.checkpoint.com > HOSTNAME=harrison-dev.checkpoint.com > LESSOPEN=|/usr/bin/lesspipe.sh %s > USER=zak > LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=4 > 0;33;01:cd=40; > 33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe > =01;32:*.com=0 > 1;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;3 > 1:*.tgz=01;31: > *.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z= > 01;31:*.gz=01; > 31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31: > *.jpg=01;35:*. > gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35: > MACHTYPE=i386 > MAIL=/var/mail/zak > LANG=en_US.UTF8 > HOST=harrison-dev.checkpoint.com > SSH_CLIENT=194.29.44.79 2642 22 > LOGNAME=zak > SHLVL=4 > GROUP=unknown > SUPPORTED=en_US.UTF8:en_US:en:en_US.iso885915 > SHELL=/bin/bash > HOSTTYPE=i386-linux > OSTYPE=linux > HOME=/root > TERM=vt100 > SSH_AUTH_SOCK=/tmp/ssh-XXXHtDKX/agent.14463 > PATH=/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin > SSH_TTY=/dev/pts/3 > OLDPWD=/tmp/zach > _=/usr/bin/env > > > > > > > > > > windows > ======================= > D:\JbuilderProjects\src\general\POI-SRC\run>set > ALLUSERSPROFILE=C:\Documents and Settings\All Users > ANT_HOME=c:\apache-ant-1.6.5 > APPDATA=C:\Documents and Settings\zak\Application Data > CLIENTNAME=Console > COMMONHOME=C:\Program Files\Rational\common > CommonProgramFiles=C:\Program Files\Common Files > COMPUTERNAME=ZAKSRV2 > ComSpec=C:\WINDOWS\system32\cmd.exe > DISPLAY=:0.0 > FORREST_HOME=c:\apache-forrest-0.7 > FP_NO_HOST_CHECK=NO > HOME=C:\Documents and Settings\zak > HOMEDRIVE=C: > HOMEPATH=\Documents and Settings\zak > HSSF.testdata.path=D:\RequiredLibs\poi\source\src\testcases\or > g\apache\poi\h > ssf\ > data\ > J2EE_HOME=c:\j2sdkee1.3.1\ > JAVA_HOME=C:\j2sdk1.4.2_08 > JAVA_HOME_1.4.1_02=C:\j2sdk1.4.1_02 > LOGONSERVER=\\HAMPSON > NUMBER_OF_PROCESSORS=1 > NUTCROOT=C:\PROGRA~1\RATIONAL\RATION~1\NUTCROOT > NUTSUFFIX=1 > NUT_SUFFIXED_SEARCHING=1 > OS=Windows_NT > Path=C:\j2sdk1.4.2_08\bin;D:\sapjcorfcLocation\;c:\apache-ant- > 1.6.5\bin;c:\a > pach > e-forrest-0.7\bin;C:\PROGRA~1\RATIONAL\RATION~1\NUTCROOT\bin;C > :\PROGRA~1\RAT > IONA > L\RATION~1\NUTCROOT\bin\x11;C:\PROGRA~1\RATIONAL\RATION~1\NUTC > ROOT\mksnt;d:\ > sapd > b\programs\bin;d:\sapdb\programs\pgm;C:\ora817\bin;C:\Program > Files\Oracle\jre\1 > .1.7\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wb > em;C:\Program > File > s\Rational\common;C:\Program Files\Rational\ClearQuest;C:\Program > Files\Rational > \Rose\TopLink\;C:\Program Files\Rational\Rational > Test;C:\JBuilder7\bin;C:\Progr > am Files\SecureCRT\;C:\PROGRA~1\ULTRAE~1 > PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH > PROCESSOR_ARCHITECTURE=x86 > PROCESSOR_IDENTIFIER=x86 Family 15 Model 2 Stepping 4, GenuineIntel > PROCESSOR_LEVEL=15 > PROCESSOR_REVISION=0204 > ProgramFiles=C:\Program Files > PROMPT=$P$G > RATL_RTHOME=C:\Program Files\Rational\Rational Test > ROOTDIR=C:\PROGRA~1\RATIONAL\RATION~1\NUTCROOT > RQAHOME=C:\Program Files\Rational\Rational Test\QualityArchitect > SAPJCORFC_LOCATION=D:\sapjcorfcLocation\ > SESSIONNAME=Console > SHELL=C:\PROGRA~1\RATIONAL\RATION~1\NUTCROOT\mksnt\sh.exe > SystemDrive=C: > SystemRoot=C:\WINDOWS > TEMP=C:\DOCUME~1\zak\LOCALS~1\Temp > TERM=nutc > TERMCAP=C:\PROGRA~1\RATIONAL\RATION~1\NUTCROOT\etc\termcap > TERMINFO=C:\PROGRA~1\RATIONAL\RATION~1\NUTCROOT\usr\lib\terminfo > TMP=C:\DOCUME~1\zak\LOCALS~1\Temp > TMPDIR=C:\DOCUME~1\zak\LOCALS~1\Temp > USERDNSDOMAIN=AD.CHECKPOINT.COM > USERDOMAIN=AD > USERNAME=zak > USERPROFILE=C:\Documents and Settings\zak > windir=C:\WINDOWS > > > > and here is the code for the test class (I built one for POI > 1.5.1 and one > for POI 2.5.1, as there is a slight difference in the POI > interfaces... > > /************************************* > ** start of v 2.5.1 class > *************************************/ > > import java.io.*; > import org.apache.poi.hssf.usermodel.*; > > public class TestWorkbook_v251 { > > public TestWorkbook_v251() { > } > > > public static void main(String[] args) { > > long[] millis = new long[2]; > > int NUM_OF_ROWS = 10; > int NUM_OF_COLS = 10; > > > // get num of rows and columns from command line > if (args != null && args.length > 0) { > NUM_OF_ROWS = Integer.parseInt(args[0]); > NUM_OF_COLS = NUM_OF_ROWS; > > if (args.length > 1) { > NUM_OF_COLS = Integer.parseInt(args[1]); > } > } > > > > TestWorkbook_v251 testLargeWorkbook1 = new TestWorkbook_v251(); > > testLargeWorkbook1.testCreateAndWriteHSSFWorkbook(NUM_OF_ROWS, > NUM_OF_COLS); > > } > > > > > public void testCreateAndWriteHSSFWorkbook(final int > NUM_OF_ROWS, final > int NUM_OF_COLS) { > try { > long[] millis = new long[4]; > millis[0] = System.currentTimeMillis(); > HSSFWorkbook hssfwb = createHSSFWorkBook(NUM_OF_ROWS, > NUM_OF_COLS); > millis[1] = System.currentTimeMillis(); > > millis[2] = System.currentTimeMillis(); > OutputStream out = getOutputStream(); > hssfwb.write(out); > millis[3] = System.currentTimeMillis(); > > System.out.println("create workBook millis=" + > (millis[1]-millis[0])); > System.out.println("write workBook millis=" + > (millis[3]-millis[2])); > System.out.println("entire test millis=" + > (millis[3]-millis[0])); > > } > catch (IOException ex) { > System.err.println("testLargeWorkbook1 failed"); > ex.printStackTrace(); > } > > } > > > > > > > private OutputStream getOutputStream() throws IOException, > FileNotFoundException { > final String fileLocation = "testXLS.xls"; > > File file = new File (fileLocation); > if (file.exists() == false) { > file.createNewFile(); > } > > BufferedOutputStream outStream = new BufferedOutputStream(new > FileOutputStream(fileLocation)); > > return outStream; > } > > public HSSFWorkbook createHSSFWorkBook(final int > NUM_OF_ROWS, final int > NUM_OF_COLS) { > HSSFWorkbook wb = new HSSFWorkbook(); > HSSFSheet hssfSheet1 = wb.createSheet("sheet1"); > > for (int i = 0; i < NUM_OF_ROWS; i++) { > HSSFRow row = hssfSheet1.createRow(i) ; > for (int j = 0; j < NUM_OF_COLS; j++) { > HSSFCell cell = row.createCell((short)j); > cell.setCellValue("cell (" + i + "," + j + ")"); > } > } > > return wb; > } > } > > /************************************* > ** end of v 2.5.1 class > *************************************/ > > > > > > /************************************* > ** start of v 1.5.1 class > *************************************/ > > import java.io.*; > import org.apache.poi.hssf.usermodel.*; > > public class TestWorkbook_v151 { > > public TestWorkbook_v151() { > } > > > public static void main(String[] args) { > > long[] millis = new long[2]; > > int NUM_OF_ROWS = 10; > int NUM_OF_COLS = 10; > > > // get num of rows and columns from command line > if (args != null && args.length > 0) { > NUM_OF_ROWS = Integer.parseInt(args[0]); > NUM_OF_COLS = NUM_OF_ROWS; > > if (args.length > 1) { > NUM_OF_COLS = Integer.parseInt(args[1]); > } > } > > > > TestWorkbook_v151 testLargeWorkbook1 = new TestWorkbook_v151(); > > testLargeWorkbook1.testCreateAndWriteHSSFWorkbook(NUM_OF_ROWS, > NUM_OF_COLS); > > } > > > > > public void testCreateAndWriteHSSFWorkbook(final int > NUM_OF_ROWS, final > int NUM_OF_COLS) { > try { > long[] millis = new long[4]; > millis[0] = System.currentTimeMillis(); > HSSFWorkbook hssfwb = createHSSFWorkBook(NUM_OF_ROWS, > NUM_OF_COLS); > millis[1] = System.currentTimeMillis(); > > millis[2] = System.currentTimeMillis(); > OutputStream out = getOutputStream(); > hssfwb.write(out); > millis[3] = System.currentTimeMillis(); > > System.out.println("create workBook millis=" + > (millis[1]-millis[0])); > System.out.println("write workBook millis=" + > (millis[3]-millis[2])); > System.out.println("entire test millis=" + > (millis[3]-millis[0])); > > } > catch (IOException ex) { > System.err.println("testLargeWorkbook1 failed"); > ex.printStackTrace(); > } > > } > > > > > > > private OutputStream getOutputStream() throws IOException, > FileNotFoundException { > final String fileLocation = "testXLS.xls"; > > File file = new File (fileLocation); > if (file.exists() == false) { > file.createNewFile(); > } > > BufferedOutputStream outStream = new BufferedOutputStream(new > FileOutputStream(fileLocation)); > > return outStream; > } > > public HSSFWorkbook createHSSFWorkBook(final int > NUM_OF_ROWS, final int > NUM_OF_COLS) { > HSSFWorkbook wb = new HSSFWorkbook(); > HSSFSheet hssfSheet1 = wb.createSheet("sheet1"); > > for (int i = 0; i < NUM_OF_ROWS; i++) { > HSSFRow row = hssfSheet1.createRow((short)i) ; > for (int j = 0; j < NUM_OF_COLS; j++) { > HSSFCell cell = row.createCell((short)j); > cell.setCellValue("cell (" + i + "," + j + ")"); > } > } > > return wb; > } > } > > > /************************************* > ** end of v 1.5.1 class > *************************************/ > > > > -----Original Message----- > From: Marx, Stefan ext. MA Fa. Lynx > [mailto:[EMAIL PROTECTED] > Sent: Thursday, August 18, 2005 2:00 PM > To: 'POI Users List' > Subject: AW: bad performance in linux > > Hi Zach, > > 'cause you didn't the machine data I assume that you measured > the times on the same machine. > After you investigated in the direction of file systems (blocksize, > journaling (Reiser, ext3), give your vm a try. Are you using a hot- > spot version. Andy Oliver from the core team wrote about huge > differences in the runtime behaviour of different flavours with SUNs > runtime set as most performant among other (Blackbird etc.). > > May be you can tune your Linux enviroment in this area. > > cu Stefan > > > -----Ursprüngliche Nachricht----- > > Von: Zach Engel [SMTP:[EMAIL PROTECTED] > > Gesendet am: Mittwoch, 17. August 2005 21:08 > > An: 'POI Users List' > > Betreff: bad performance in linux > > > > Hi > > > > Need some help with the following performance problem: > > > > I have some code that creates a HSSFWorkBook > > (about 400 rows of 10 cells each) and then writes it > > to an output stream (out = HttpServletResponse.getOutputStream()) > > > > This takes 2 seconds on a WINDOWS machine. > > Here is some info from my log > > [17 Aug 2005 18:12:30 DEBUG] - hssf entire write millis=1908 > > [17 Aug 2005 18:12:30 DEBUG] - hssf getbytes millis=497 > > [17 Aug 2005 18:12:30 DEBUG] - hssf createDocument millis=713 > > [17 Aug 2005 18:12:30 DEBUG] - hssf writeFilesystem millis=698 > > > > When I run the same code on a linux machine (Linux 2.4.20-13.7), > > it takes 7-10 seconds. > > Here is some info from my log > > [2005-08-17 19:07:21 DEBUG] - hssf entire write millis=7051 > > [2005-08-17 19:07:21 DEBUG] - hssf getbytes millis=4441 > > [2005-08-17 19:07:21 DEBUG] - hssf createDocument millis=46 > > [2005-08-17 19:07:21 DEBUG] - hssf writeFilesystem millis=2562 > > > > After downloading the most current POI sources, and > debugging a little, > > I discovered that when the getBytes() method (in > HSSFWorkbook.write) tries > > to serialize the SSTRecord record, it takes about 4000 > milliseconds, > > While it takes only about 47 milliseconds on windows > > > > > > In addition to that, while investigating the > writeFilesystem method, > > I discovered that most of the time is spent writing block2 > > (indeed shhet2 contains the 400 rows of data) > > > > From the Linux log: > > [2005-08-17 19:28:02 DEBUG] - hssf writeBlock[1] millis=0 > > [2005-08-17 19:28:04 DEBUG] - hssf writeBlock[2] millis=2300 > > [2005-08-17 19:28:05 DEBUG] - hssf writeBlock[3] millis=210 > > [2005-08-17 19:28:05 DEBUG] - hssf writeBlock[4] millis=0 > > [2005-08-17 19:28:05 DEBUG] - hssf writeBlock[5] millis=0 > > [2005-08-17 19:28:05 DEBUG] - hssf writeBlock[6] millis=0 > > [2005-08-17 19:28:05 DEBUG] - hssf writeBlock[7] millis=0 > > [2005-08-17 19:28:05 DEBUG] - hssf writeBlock[8] millis=1 > > > > > > From the WINDOWS log: > > [17 Aug 2005 18:33:31 DEBUG] - hssf writeBlock[1] millis=0 > > [17 Aug 2005 18:33:32 DEBUG] - hssf writeBlock[2] millis=1807 > > [17 Aug 2005 18:33:32 DEBUG] - hssf writeBlock[3] millis=0 > > [17 Aug 2005 18:33:32 DEBUG] - hssf writeBlock[4] millis=0 > > [17 Aug 2005 18:33:32 DEBUG] - hssf writeBlock[5] millis=0 > > [17 Aug 2005 18:33:32 DEBUG] - hssf writeBlock[6] millis=0 > > [17 Aug 2005 18:33:32 DEBUG] - hssf writeBlock[7] millis=0 > > [17 Aug 2005 18:33:32 DEBUG] - hssf writeBlock[8] millis=0 > > > > > > > > > > > > > > One last thing regarding the LANG environment variable: > > While looking for answers on the web I noticed there have > been some issues > > with encoding settings. > > My LANG on the linux is set to en_US.UTF8 > > I changed it to ISO-8859-1, and then ran the tomcat, > > but the results where the same. > > > > I hope you can offer some kind of explanation to the wide difference > > between > > the 2 environments. > > > > Thank you > > > > Zach Engel > > Check Point > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > Mailing List: http://jakarta.apache.org/site/mail2.html#poi > > The Apache Jakarta Poi Project: http://jakarta.apache.org/poi/ > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > Mailing List: http://jakarta.apache.org/site/mail2.html#poi > The Apache Jakarta Poi Project: http://jakarta.apache.org/poi/ > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > Mailing List: http://jakarta.apache.org/site/mail2.html#poi > The Apache Jakarta Poi Project: http://jakarta.apache.org/poi/ > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] Mailing List: http://jakarta.apache.org/site/mail2.html#poi The Apache Jakarta Poi Project: http://jakarta.apache.org/poi/ --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] Mailing List: http://jakarta.apache.org/site/mail2.html#poi The Apache Jakarta Poi Project: http://jakarta.apache.org/poi/
