Hi everyone,
Currently, I'm working on setting a cluster using amazon ec2. We are using
Meep-mpi with OpenMPI. We are doing some testing with exercises and we are
having some issues with the following exercise *3d_ring_transmission.ctl*
Our problem is that the simulation doesn't improve the time duration as we
increase the numbers of nodes of the cluster, I mean, with 2,4,6,... nodes,
the time instead of decrease is increasing. We don't know exactly what
happens. We suppose like the exercise does a lot of "writings" ((to-appended
"hz" (at-every 0.5 output-hfield-z))), it takes more time, because of the
AmazonEC2 nodes virtualization. But We are not sure about this. The final
output is 20 GB.
Also, I paste two outputs running with 2 and 4 nodes. These outputs are just
the beggining not the entire, as you can see in the outputs the time between
each step actually is better with 4 nodes than 2 but not in all, in the
timesteps 1,41,81,... the time is less with 2 nodes than with 4 nodes. And
in the end, the time is better with 2 nodes =/
I'm not related to FDTD simulations, in fact I'm related to computer science
field, but I would like to understand a little better about this. My
question is: my assuption is correct about the virtualization? can we use
the ram memory for the "writings" (at-every 0.5) and in the end write to the
hard drive?
For run the ctl is:
meep no-ring?=true 3d_ring_transmission.ctl | tee transmission0.out
thanks!
*
*
*3d_ring_transmission.ctl*
;Simulation of a 3D ring resonator in add-drop configuration
;OUTPUT: Transmission flux from thru port and drop port
;General parameters
(define-param nc 3.03) ;core refractive index
(define-param ns 1.67) ;substrate refractive index
(define-param r1 2) ;inner radius of ring in microns
(define-param r2 2.5) ;outer radius of ring
(define-param w 0.55) ;waveguide width in microns
(define-param gap 0.2) ; gap between ring and straight waveguide in microns
(define-param pad 0.75) ;padding distance in microns
(define-param h 0.405) ; height of waveguides in microns
(define-param hs 0.6) ;height of substrate
(define-param dpml 1) ;pml thickness
;Cell structure definition
(define sx (* 2 (+ r2 gap w pad dpml))) ; X direction cell size
;(define sy (* 2 (+ r2 pad dpml))) ; Y direction cell size
(define sy (* 2 (+ r2 pad ))) ; Y direction cell size
(define sz (* 2 (+ hs dpml))) ; Z direction cell size
(set! geometry-lattice (make lattice (size sx sy sz)))
;FLAG to denote complete structure = false, waveguide only = true
(define-param no-ring? true) ;default simulate waveguides only
;Construct waveguide and ring resonator
(define subs-width (* 2 (+ r2 gap w pad)))
(define subs-length (* 2 (+ r2 pad)))
(if no-ring?
(begin
(set! geometry (list
;substrate
(make block
(center 0 0 (* -1 (/ hs 2))) ; center of substrate
;(size (+ subs-width dpml) (+ subs-length
dpml) hs) ; dimension of substrate is (wxLxh) = 2*(r2+gap+w+pad) x
2*(r2+pad) x hs
(size subs-width subs-length hs)
(material (make dielectric (index ns))))
;waveguides
(make block
(center (+ r2 gap (/ w 2)) 0 (/ h 2))
;right waveguide
;(size w (+ subs-length dpml) h)
(size w subs-length h)
(material (make dielectric (index nc))))
(make block
(center (* -1 (+ r2 gap (/ w 2))) 0 (/ h 2))
;(size w (+ subs-length dpml) h)
(size w subs-length h)
(material (make dielectric (index nc))))
)))
;else condition
(begin
(set! geometry (list
;substrate
(make block
(center 0 0 (* -1 (/ hs 2))) ; center of substrate
;(size (+ subs-width dpml) (+ subs-length
dpml) hs) ; dimension of substrate is (wxLxh) = 2*(r2+gap+w+pad) x
2*(r2+pad) x hs
(size subs-width subs-length hs)
(material (make dielectric (index ns))))
;waveguides
(make block
(center (+ r2 gap (/ w 2)) 0 (/ h 2))
;right waveguide
;(size w (+ subs-length dpml)h h)
(size w subs-length h)
(material (make dielectric (index nc))))
(make block
(center (* -1 (+ r2 gap (/ w 2))) 0 (/ h 2))
;(size w (+ subs-length dpml) h)
(size w subs-length h)
(material (make dielectric (index nc))))
;ring resonator structure here
(make cylinder
(center 0 0 (/ h 2) ) (height h)
(radius r2) (material (make dielectric (index nc))))
(make cylinder
(center 0 0 (/ h 2) ) (height h)
(radius r1) (material air))
)))
)
;PML setup
(set! pml-layers (list (make pml (thickness dpml))))
;RESOLUTION
(set-param! resolution 40) ; use resolution, 40 for a 3.8 GB pC
;Source definition
(define-param fcen 0.56) ; 0.56 center frequency at 170 Thz
(define-param df 0.4) ; 0.2 pulse width 60 Thz
(define-param nfreq 2000) ; 200 number of freq elements to compute flux
(set! sources (list
(make source
(src (make gaussian-src (frequency fcen) (fwidth df)))
(component Hz) ; TE excitation
;(center (* -1 (+ r2 gap (/ w 2))) (+ r2 (/ pad
2)) (/ h 2))
(center (* -1 (+ r2 gap (/ w 2))) r2 (/ h 2))
(size w 0 h )))) ;point source
;Define flux computation Region
(define transmission_in ;input transmission
(add-flux fcen df nfreq
(make flux-region
(center (* -1 (+ r2 gap (/ w 2))) r1 (/ h 2))
(size (* w 1.0) 0 (* h 1.0))
(weight -1.0))))
(define transmission_thru ;transmitted flux thru port
(add-flux fcen df nfreq
(make flux-region
; (center (* -1 (+ r2 gap (/ w 2))) (+ r2 (/ pad 2))
(/ h 2)) ;ERROR HERE
;(center (* -1 (+ r2 gap (/ w 2))) (* -1 (+ r2 (/
pad 2))) (/ h 2))
(center (* -1 (+ r2 gap (/ w 2))) (* -1 r1) (/ h 2))
(size (* w 1.0) 0 (* h 1.0))
(weight -1.0)))) ; because flux is outward the
waveguide (negative axis)
(define transmission_drop ;transmitted flux drop port
(add-flux fcen df nfreq
(make flux-region
;(center (+ r2 gap (/ w 2)) (+ r2 (/ pad 2)) (/ h 2))
(center (* +1 (+ r2 gap (/ w 2))) r1 (/ h 2))
(size (* w 1.0) 0 (* h 1.0))
(weight +1.0)))) ; because flux is outward the
waveguide (positive axis)
;Run command
(run-sources+
(stop-when-fields-decayed 50 Hz
(vector3 (* -1 (+ r2 gap (/ w 2))) (* -1 r1) (/ h 2))
1e-4)
(to-appended "hz" (at-every 0.5 output-hfield-z))
(at-beginning output-epsilon)
)
(display-fluxes transmission_in transmission_thru transmission_drop) ;
print out flux spectrum
*Output 2 nodes*
Using MPI version 2.0, 2 processes
command-line param: no-ring?=true
-----------
Initializing structure...
Working in 3D dimensions.
Computational cell is 10 x 6.5 x 3.2 with resolution 40
block, center = (0,0,-0.3)
size (8,6.5,0.6)
axes (1,0,0), (0,1,0), (0,0,1)
dielectric constant epsilon diagonal = (2.7889,2.7889,2.7889)
block, center = (2.975,0,0.2025)
size (0.55,6.5,0.405)
axes (1,0,0), (0,1,0), (0,0,1)
dielectric constant epsilon diagonal = (9.1809,9.1809,9.1809)
block, center = (-2.975,0,0.2025)
size (0.55,6.5,0.405)
axes (1,0,0), (0,1,0), (0,0,1)
dielectric constant epsilon diagonal = (9.1809,9.1809,9.1809)
command-line param: no-ring?=true
subpixel-averaging is 70.7238% done, 1.65643 s remaining
subpixel-averaging is 65.895% done, 2.07103 s remaining
subpixel-averaging is 70.5009% done, 1.67494 s remaining
time for set_epsilon = 67.9664 s
-----------
creating output file "./3d-hz.h5"...
creating output file "./3d-eps-000000.00.h5"...*on time step 1
(time=0.0125), 55.2413 s/step*
on time step 5 (time=0.0625), 1.32752 s/step
on time step 9 (time=0.1125), 1.32443 s/step
on time step 13 (time=0.1625), 1.32885 s/step
on time step 16 (time=0.2), 1.35196 s/step
on time step 19 (time=0.2375), 1.46996 s/step
on time step 22 (time=0.275), 1.58233 s/step
on time step 25 (time=0.3125), 1.45481 s/step
on time step 28 (time=0.35), 1.53447 s/step
on time step 31 (time=0.3875), 1.52081 s/step
on time step 34 (time=0.425), 1.49163 s/step
on time step 37 (time=0.4625), 1.47459 s/step
on time step 40 (time=0.5), 3.70836 s/step*on time step 41
(time=0.5125), 74.9581 s/step*
on time step 44 (time=0.55), 1.43735 s/step
on time step 47 (time=0.5875), 1.40395 s/step
on time step 50 (time=0.625), 1.40748 s/step
on time step 53 (time=0.6625), 1.42533 s/step
on time step 56 (time=0.7), 1.39888 s/step
on time step 59 (time=0.7375), 1.35495 s/step
on time step 62 (time=0.775), 1.34053 s/step
on time step 65 (time=0.8125), 1.3636 s/step
on time step 68 (time=0.85), 1.35887 s/step
on time step 71 (time=0.8875), 1.37078 s/step
on time step 74 (time=0.925), 1.34815 s/step
on time step 78 (time=0.975), 1.31821 s/step
on time step 80 (time=1), 5.4548 s/step
*on time step 81 (time=1.0125), 75.7812 s/step** *
*Output 4 nodes*
Using MPI version 2.0, 4 processes
command-line param: no-ring?=true
command-line param: no-ring?=true
command-line param: no-ring?=true
command-line param: no-ring?=true
-----------
Initializing structure...
Working in 3D dimensions.
Computational cell is 10 x 6.5 x 3.2 with resolution 40
block, center = (0,0,-0.3)
size (8,6.5,0.6)
axes (1,0,0), (0,1,0), (0,0,1)
dielectric constant epsilon diagonal = (2.7889,2.7889,2.7889)
block, center = (2.975,0,0.2025)
size (0.55,6.5,0.405)
axes (1,0,0), (0,1,0), (0,0,1)
dielectric constant epsilon diagonal = (9.1809,9.1809,9.1809)
block, center = (-2.975,0,0.2025)
size (0.55,6.5,0.405)
axes (1,0,0), (0,1,0), (0,0,1)
dielectric constant epsilon diagonal = (9.1809,9.1809,9.1809)
time for set_epsilon = 29.093 s
-----------
creating output file "./3d-hz.h5"...
creating output file "./3d-eps-000000.00.h5"...cd ..*on time step 1
(time=0.0125), 54.5664 s/step*
on time step 7 (time=0.0875), 0.686385 s/step
on time step 13 (time=0.1625), 0.672188 s/step
on time step 20 (time=0.25), 0.65516 s/step
on time step 27 (time=0.3375), 0.655888 s/step
on time step 34 (time=0.425), 0.658867 s/step
on time step 40 (time=0.5), 1.42186 s/step*on time step 41
(time=0.5125), 114.796 s/step*
on time step 48 (time=0.6), 0.652713 s/step
on time step 55 (time=0.6875), 0.65162 s/step
on time step 61 (time=0.7625), 0.69854 s/step
on time step 68 (time=0.85), 0.646802 s/step
on time step 75 (time=0.9375), 0.658464 s/step
on time step 80 (time=1), 1.59971 s/step
*on time step 81 (time=1.0125), 110.083 s/step*
--
- Markos
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It's not what you know, it's what you can learn...
_______________________________________________
meep-discuss mailing list
meep-discuss@ab-initio.mit.edu
http://ab-initio.mit.edu/cgi-bin/mailman/listinfo/meep-discuss
