Simon: I have made a few optimizations regarding memory management in EPS. In your case, these changes will allocate 1 vector less (maybe 2). If you are using the repository version, just pull and try again. Otherwise, wait until slepc-3.7.2 is released (in a few days). Jose
> El 16 jul 2016, a las 17:00, Barry Smith <[email protected]> escribió: > > > Send configure.log to [email protected] > > >> On Jul 16, 2016, at 8:40 AM, Simon Burton <[email protected]> wrote: >> >> >> Hi again, >> >> I found another machine with enough ram to run this (i think). >> >> Running into another problem now, with dgemv: >> >> [0] EPSSetUp_Power(): Warning: parameter mpd ignored >> [0] STSetUp(): Setting up new ST >> Intel MKL ERROR: Parameter 6 was incorrect on entry to DGEMV . >> [0] BV_SafeSqrt(): Zero norm, either the vector is zero or a semi-inner >> product is being used >> >> >> I dug into this in gdb a bit: >> >> >> Breakpoint 2, 0x00007ffff4f4cbd0 in dgemv_ () >> from >> /usr/physics/ic15/composer_xe_2015.0.090/mkl/lib/intel64/libmkl_intel_lp64.so >> (gdb) bt >> #0 0x00007ffff4f4cbd0 in dgemv_ () from >> /usr/physics/ic15/composer_xe_2015.0.090/mkl/lib/intel64/libmkl_intel_lp64.so >> #1 0x00007ffff5e14b4b in BVDotVec_BLAS_Private (bv=0x6ba6b0, n_=4294967296, >> k_=1, A=0x7fe7f23b3650, x=0x7fe7f23b3650, >> y=0x75a3b0, mpi=PETSC_FALSE) at >> /suphys/sburton/local/slepc-3.7.1/src/sys/classes/bv/interface/bvblas.c:274 >> #2 0x00007ffff5dcbd86 in BVDotVec_Svec (X=0x6ba6b0, y=0x74dbc0, m=0x75a3b0) >> at >> /suphys/sburton/local/slepc-3.7.1/src/sys/classes/bv/impls/svec/svec.c:150 >> #3 0x00007ffff5dffd58 in BVDotVec (X=0x6ba6b0, y=0x74dbc0, m=0x75a3b0) >> at >> /suphys/sburton/local/slepc-3.7.1/src/sys/classes/bv/interface/bvglobal.c:191 >> #4 0x00007ffff5e1aad9 in BVOrthogonalizeCGS1 (bv=0x6ba6b0, j=0, v=0x0, >> H=0x75a3b0, onorm=0x7fffffffdc28, >> norm=0x7fffffffdc20) at >> /suphys/sburton/local/slepc-3.7.1/src/sys/classes/bv/interface/bvorthog.c:81 >> #5 0x00007ffff5e1c1bb in BVOrthogonalizeCGS (bv=0x6ba6b0, j=0, v=0x0, >> H=0x0, norm=0x7fffffffddb0, lindep=0x7fffffffddac) >> at >> /suphys/sburton/local/slepc-3.7.1/src/sys/classes/bv/interface/bvorthog.c:214 >> #6 0x00007ffff5e1ddfd in BVOrthogonalizeColumn (bv=0x6ba6b0, j=0, H=0x0, >> norm=0x7fffffffddb0, lindep=0x7fffffffddac) >> at >> /suphys/sburton/local/slepc-3.7.1/src/sys/classes/bv/interface/bvorthog.c:371 >> #7 0x00007ffff6050986 in EPSGetStartVector (eps=0x6a3ee0, i=0, >> breakdown=0x0) >> at /suphys/sburton/local/slepc-3.7.1/src/eps/interface/epssolve.c:758 >> #8 0x00007ffff5f52812 in EPSSolve_Power (eps=0x6a3ee0) at >> /suphys/sburton/local/slepc-3.7.1/src/eps/impls/power/power.c:103 >> #9 0x00007ffff6049b28 in EPSSolve (eps=0x6a3ee0) at >> /suphys/sburton/local/slepc-3.7.1/src/eps/interface/epssolve.c:101 >> #10 0x0000000000401430 in main () >> (gdb) up >> #1 0x00007ffff5e14b4b in BVDotVec_BLAS_Private (bv=0x6ba6b0, n_=4294967296, >> k_=1, A=0x7fe7f23b3650, x=0x7fe7f23b3650, >> y=0x75a3b0, mpi=PETSC_FALSE) at >> /suphys/sburton/local/slepc-3.7.1/src/sys/classes/bv/interface/bvblas.c:274 >> 274 if (n) >> PetscStackCallBLAS("BLASgemv",BLASgemv_("C",&n,&k,&done,A,&n,x,&one,&zero,y,&one)); >> (gdb) print n >> $1 = 4294967296 >> (gdb) print sizeof(n) >> $2 = 8 >> (gdb) step >> Intel MKL ERROR: Parameter 6 was incorrect on entry to DGEMV . >> >> >> It looks to me like slepc is doing it right, but with error messages >> like this who knows. It's a bit beyond me debugging assembly. >> >> Originally I built petsc with --download-fblaslapack but i don't think >> it was working with 64bit indexes (?) >> >> Maybe I should try another blas. >> >> Simon. >> >> >> On Sat, 16 Jul 2016 07:17:44 +1000 >> Simon Burton <[email protected]> wrote: >> >>> On Fri, 15 Jul 2016 19:53:31 +0200 >>> "Jose E. Roman" <[email protected]> wrote: >>> >>>> >>>> The default spectral transformation (STSHIFT) will allocate just one >>>> vector. At which exact point are you seeing that it allocates a bunch of >>>> vectors? >>> >>> Yes I think you are right. >>> I can get beyond STSetUp with the right settings. >>> Now the solver runs out of memory inside EPSGetStartVector. >>> >>>> >>>> Is this the unmodified ex3.c? Or did you change anything like >>>> EPSSetOperators(eps,A,B) ? >>> >>> good question. I didn't change much, let me try again the original. >>> >>>> Do you get the same behaviour with the original ex3 with the same problem >>>> size? >>> >>> Yes >>> >>>> >>>> Do you have the same problem with a smaller problem? (half size, say) >>> >>> Halving n gives a quarter of the dimension, which is 8gb vector sizes. >>> It works fine and uses a total of 48gb ram. Oh, I see at one point during >>> initialization it hits a maximum of 56gb. >>> >>> So I guess it needs to keep 6 vectors in total. >>> With the original problem size this becomes 192gb which is >>> just a few gb too much to crunch. I guess I can still try it, >>> but it doesn't feel good hitting the harddrive that much. >>> >>> Thanks for the suggestions. >>> >>> Simon. >
