I would like to get the transmission spectrum for a structure with: cross-section=7x7; length=20; resolution=160; =4.0141e+09 points
Is this possible? What are the expected resource requirements? What is the expected runtime? How can it be optimized? I seem to get the following error: meep: Cannot split -76420096 grid points into 2 parts application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3 Searching for this error in the newsgroup I found: http://thread.gmane.org/gmane.comp.science.electromagnetism.meep.general/511 /focus=539 which refers to 1.3068e+16 points as excessive. In an effort to determine the limit I have run a 3D version of holey-wvg-cavity.cpp with: cross-section=3x3; length=10; resolution=40; =5.7600e+06 points This takes overnight (about 12 hours) using: MPICH2 on CentOS 5.5 2 x Intel Xeon W3540 2.93GHz (=8), 9 GB RAM Using MPI version 2.1, Meep 1.0.3, Using libctl 3.1 and Guile 1.8.0. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28847 ___ 27 2 518m 416m 9.9m R 99.2 4.8 2:25.23 3dholey-wvg-cav 28848 ___ 27 2 523m 421m 10m R 99.2 4.9 2:26.10 3dholey-wvg-cav 28850 ___ 27 2 471m 367m 8016 R 99.2 4.2 2:25.36 3dholey-wvg-cav 28851 ___ 27 2 542m 439m 9.8m R 99.2 5.1 2:25.78 3dholey-wvg-cav 28844 ___ 27 2 615m 512m 9.8m R 97.3 5.9 2:25.67 3dholey-wvg-cav 28845 ___ 27 2 523m 421m 10m R 97.3 4.9 2:25.43 3dholey-wvg-cav 28846 ___ 27 2 518m 416m 10m R 97.3 4.8 2:25.50 3dholey-wvg-cav 28849 ___ 27 2 472m 368m 7708 R 97.3 4.3 2:25.59 3dholey-wvg-cav 29028 ___ 15 0 10884 1020 672 R 1.9 0.0 0:00.03 top (about 0.7 s/step) When I increase the resolution to 160 the memory is exceeded resulting in numerous "subpixel-averaging" output lines followed by: rank 2 in job 1 xeon_53146 caused collective abort of all ranks exit status of rank 2: killed by signal 9 Thus I am looking into using: Open MPI on Solaris 10 Sun SPARC Enterprise M4000, 4x4x2 x 2.4 GHz SPARC64 VII (=32), 32 GB RAM Using MPI version 2.0, Meep 1.1.1, Using libctl 3.1 and Guile 1.6.7. On which I always get the following compilation warnings: "/usr/local/meep-mpi/include/meep/vec.hpp", line 894: Warning: S hides meep::symmetry::S. "/usr/local/meep-mpi/include/meep.hpp", line 826: Warning: gv hides meep::fields_chunk::gv. "/usr/local/meep-mpi/include/meep.hpp", line 958: Warning: k hides meep::fields::k. Continuing for the specifications of: cross-section=3x3; length=10; resolution=40; =5.7600e+06 points I have found a quick rule of thumb to be to choose the number of processes to be slightly lower than the number of cores on a machine. When using more processes the time for set_epsilon is quicker, the first step takes longer while the time for subsequent steps are as follows on this machine with more memory but seemingly not much faster: -np 1 2.57799 s/step -np 4 0.89036 s/step -np 16 0.682598 s/step -np 20 0.673163 s/step -np 24 0.687442 s/step -np 32 0.692809 s/step (corresponds to the below) -np 64 7.13687 s/step (more *processes* than cores, cores are multi-*threading*) PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 4749 ___ 650M 96M cpu17 0 0 0:05:52 3.1% 3dholey-wvg-cav/1 4745 ___ 658M 104M cpu22 0 0 0:05:56 3.1% 3dholey-wvg-cav/1 4743 ___ 658M 105M cpu14 0 0 0:05:52 3.1% 3dholey-wvg-cav/1 4739 ___ 658M 105M cpu7 0 0 0:05:55 3.1% 3dholey-wvg-cav/1 4737 ___ 666M 113M cpu4 0 0 0:05:57 3.1% 3dholey-wvg-cav/1 4735 ___ 666M 113M cpu1 0 0 0:05:55 3.1% 3dholey-wvg-cav/1 4733 ___ 658M 105M cpu25 0 0 0:05:56 3.1% 3dholey-wvg-cav/1 4731 ___ 658M 105M cpu8 0 0 0:05:54 3.1% 3dholey-wvg-cav/1 4729 ___ 666M 113M cpu0 0 0 0:05:56 3.1% 3dholey-wvg-cav/1 4727 ___ 666M 113M cpu5 0 0 0:05:56 3.1% 3dholey-wvg-cav/1 4719 ___ 666M 113M cpu29 0 0 0:05:54 3.1% 3dholey-wvg-cav/1 4717 ___ 658M 105M cpu11 0 0 0:05:54 3.1% 3dholey-wvg-cav/1 4715 ___ 658M 105M cpu26 0 0 0:05:54 3.1% 3dholey-wvg-cav/1 4713 ___ 666M 113M cpu20 0 0 0:05:52 3.1% 3dholey-wvg-cav/1 4709 ___ 658M 105M cpu6 0 0 0:05:56 3.1% 3dholey-wvg-cav/1 4703 ___ 666M 113M cpu3 0 0 0:05:53 3.1% 3dholey-wvg-cav/1 4701 ___ 702M 149M cpu13 0 0 0:05:55 3.1% 3dholey-wvg-cav/1 4693 ___ 650M 97M cpu30 0 0 0:05:55 3.1% 3dholey-wvg-cav/1 4695 ___ 670M 117M cpu9 0 0 0:05:56 3.1% 3dholey-wvg-cav/1 4707 ___ 658M 105M cpu2 0 0 0:05:57 3.1% 3dholey-wvg-cav/1 4711 ___ 666M 113M cpu27 0 0 0:05:56 3.1% 3dholey-wvg-cav/1 4687 ___ 662M 109M cpu12 0 0 0:05:53 3.1% 3dholey-wvg-cav/1 4697 ___ 670M 117M cpu10 0 0 0:05:57 3.1% 3dholey-wvg-cav/1 4747 ___ 650M 96M cpu28 0 0 0:05:53 3.1% 3dholey-wvg-cav/1 4725 ___ 658M 105M cpu18 0 0 0:05:55 3.1% 3dholey-wvg-cav/1 4723 ___ 658M 105M cpu15 0 0 0:05:53 3.1% 3dholey-wvg-cav/1 4689 ___ 658M 105M cpu19 0 0 0:05:51 3.1% 3dholey-wvg-cav/1 4691 ___ 650M 97M cpu31 0 0 0:05:50 3.1% 3dholey-wvg-cav/1 4699 ___ 698M 145M cpu24 0 0 0:05:52 3.1% 3dholey-wvg-cav/1 4721 ___ 666M 113M cpu23 0 0 0:05:54 3.1% 3dholey-wvg-cav/1 4741 ___ 658M 105M run 0 0 0:05:54 3.1% 3dholey-wvg-cav/1 4705 ___ 666M 113M cpu21 0 0 0:05:55 3.1% 3dholey-wvg-cav/1 4501 ___ 6160K 3392K cpu16 59 0 0:00:03 0.0% prstat/1 Testing the specifications of: cross-section=7x7; length=20; resolution=40; =6.2720e+7 points I get the following response: Using MPI version 2.0, 4 processes holey wg transmission tests... subpixel-averaging is 34.1141% done, 7.72685 s remaining subpixel-averaging is 67.8073% done, 1.89966 s remaining subpixel-averaging is 33.8135% done, 7.832 s remaining subpixel-averaging is 67.5788% done, 1.91902 s remaining subpixel-averaging is 33.8014% done, 7.83423 s remaining subpixel-averaging is 67.3865% done, 1.93622 s remaining time for set_epsilon = 85.5157 s subpixel-averaging is 34.535% done, 7.58435 s remaining subpixel-averaging is 68.8414% done, 1.81079 s remaining subpixel-averaging is 34.583% done, 7.56884 s remaining subpixel-averaging is 68.8535% done, 1.8098 s remaining subpixel-averaging is 33.4527% done, 7.95833 s remaining subpixel-averaging is 66.4365% done, 2.0214 s remaining subpixel-averaging is 99.8892% done, 0.00443559 s remaining time for set_epsilon = 84.8568 s Output directory 3dholey-wvg-cavity-out already exists! Output directory 3dholey-wvg-cavity-out-0 already exists! Output directory 3dholey-wvg-cavity-out-1 already exists! Output directory 3dholey-wvg-cavity-out-2 already exists! Using output directory 3dholey-wvg-cavity-out-3/ Using output directory 3dholey-wvg-cavity-out-3/ creating output file "3dholey-wvg-cavity-out-3/eps-000000.00.h5"... creating output file "3dholey-wvg-cavity-out-3/eps-000000.00.h5"... volume sources added... simulating structure with and without holes... [m4k:04440] *** Process received signal *** [m4k:04440] Signal: Abort (6) [m4k:04440] Signal code: (-1) /opt/SUNWhpc/HPC8.1/sun/lib/libopen-pal.so.0.0.0:0x55c7c /lib/libc.so.1:0xc8dc8 /lib/libc.so.1:0xbd460 /lib/libc.so.1:0xbd648 /lib/libc.so.1:__lwp_kill+0x8 [ Signal -53127840 (?)] /lib/libc.so.1:abort+0xd0 /opt/SUNWhpc/HPC8.1/sun/lib/libCrun.so.1:__1cH__CimplRdefault_terminate6F_v_ +0x4 /opt/SUNWhpc/HPC8.1/sun/lib/libCrun.so.1:__1cH__CimplMex_terminate6F_v_+0x2c /opt/SUNWhpc/HPC8.1/sun/lib/libCrun.so.1:0x5b04 /opt/SUNWhpc/HPC8.1/sun/lib/libCrun.so.1:__1cG__CrunRex_chk_unexpected6F_v_+ 0xa4 /opt/SUNWhpc/HPC8.1/sun/lib/libCrun.so.1:__1c2n6FI_pv_+0x94 /3dholey-wvg-cavity.dac:__1cEmeepMfields_chunkHstep_db6Mn0AKfield_type__b_+0 x3c4 /3dholey-wvg-cavity.dac:__1cEmeepGfieldsHstep_db6Mn0AKfield_type__v_+0x34 /3dholey-wvg-cavity.dac:__1cEmeepGfieldsEstep6M_v_+0x18c /3dholey-wvg-cavity.dac:main+0xd64 /3dholey-wvg-cavity.dac:_start+0x108 [m4k:04440] *** End of error message *** [m4k:04438] *** Process received signal *** [m4k:04438] Signal: Abort (6) [m4k:04438] Signal code: (-1) /opt/SUNWhpc/HPC8.1/sun/lib/libopen-pal.so.0.0.0:0x55c7c /lib/libc.so.1:0xc8dc8 /lib/libc.so.1:0xbd460 /lib/libc.so.1:0xbd648 /lib/libc.so.1:__lwp_kill+0x8 [ Signal -53127840 (?)] /lib/libc.so.1:abort+0xd0 /opt/SUNWhpc/HPC8.1/sun/lib/libCrun.so.1:__1cH__CimplRdefault_terminate6F_v_ +0x4 /opt/SUNWhpc/HPC8.1/sun/lib/libCrun.so.1:__1cH__CimplMex_terminate6F_v_+0x2c /opt/SUNWhpc/HPC8.1/sun/lib/libCrun.so.1:0x5b04 /opt/SUNWhpc/HPC8.1/sun/lib/libCrun.so.1:__1cG__CrunRex_chk_unexpected6F_v_+ 0xa4 /opt/SUNWhpc/HPC8.1/sun/lib/libCrun.so.1:__1c2n6FI_pv_+0x94 /3dholey-wvg-cavity.dac:__1cEmeepMfields_chunkHstep_db6Mn0AKfield_type__b_+0 x3c4 /3dholey-wvg-cavity.dac:__1cEmeepGfieldsHstep_db6Mn0AKfield_type__v_+0x34 /3dholey-wvg-cavity.dac:__1cEmeepGfieldsEstep6M_v_+0x18c /3dholey-wvg-cavity.dac:main+0xd64 /3dholey-wvg-cavity.dac:_start+0x108 [m4k:04438] *** End of error message *** -------------------------------------------------------------------------- mpirun noticed that process rank 1 with PID 4438 on node carlm4k exited on signal 6 (Abort). -------------------------------------------------------------------------- Thanks in advance for advice _______________________________________________ meep-discuss mailing list [email protected] http://ab-initio.mit.edu/cgi-bin/mailman/listinfo/meep-discuss
