Anecdotes on my OOM error: It only fails on one test, it is *sort of* working though.
I had the NVIDIA monitor up while I run the tests. I did this a few times and monitored results (if anyone knows a CLI way to do this and pipe results to a file- dope). In general, the background memory utilization of my graphics card is 33% (of 1G total). As tests run, utilization is as high as 50% of memory and 99% of GPU utilization (for other tests). This makes the OOM error even more curious. Does anyone know a good util to use to monitor the card while it is running. Keep up the fight Andrew, I feel like this is very close. tg Trevor Grant Data Scientist https://github.com/rawkintrevo http://stackexchange.com/users/3002022/rawkintrevo http://trevorgrant.org *"Fortunate is he, who is able to know the causes of things." -Virgil* On Fri, Oct 28, 2016 at 6:01 PM, Trevor Grant <[email protected]> wrote: > I was finally able to hit an OOM error. > > I initially was running everything from the top. > > When I run mvn test from the viennacl directory... > > > /mahout/viennacl$ mvn test > [INFO] Scanning for projects... > [WARNING] > [WARNING] Some problems were encountered while building the effective > model for org.apache.mahout:mahout-native-viennacl_2.10:jar:0.13. > 0-SNAPSHOT > [WARNING] 'artifactId' contains an expression but should be a constant. @ > org.apache.mahout:mahout-native-viennacl_${scala.compat.version}:[unknown-version], > /home/trevor/gits/mahout/viennacl/pom.xml, line 31, column 15 > [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but > found duplicate declaration of plugin org.codehaus.mojo:exec-maven-plugin > @ org.apache.mahout:mahout-native-viennacl_${scala. > compat.version}:[unknown-version], /home/trevor/gits/mahout/viennacl/pom.xml, > line 181, column 15 > [WARNING] > [WARNING] It is highly recommended to fix these problems because they > threaten the stability of your build. > [WARNING] > [WARNING] For this reason, future Maven versions might no longer support > building such malformed projects. > [WARNING] > [INFO] > > [INFO] ------------------------------------------------------------ > ------------ > [INFO] Building Mahout Native VienniaCL OpenCL Bindings 0.13.0-SNAPSHOT > [INFO] ------------------------------------------------------------ > ------------ > [INFO] > [INFO] --- maven-enforcer-plugin:1.4:enforce (enforce-versions) @ > mahout-native-viennacl_2.10 --- > [INFO] > [INFO] --- scala-maven-plugin:3.2.0:add-source (add-scala-sources) @ > mahout-native-viennacl_2.10 --- > [INFO] Add Source directory: /home/trevor/gits/mahout/ > viennacl/src/main/scala > [INFO] Add Test Source directory: /home/trevor/gits/mahout/ > viennacl/src/test/scala > [INFO] > [INFO] --- maven-dependency-plugin:2.3:properties (default) @ > mahout-native-viennacl_2.10 --- > [INFO] > [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ > mahout-native-viennacl_2.10 --- > [INFO] > [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ > mahout-native-viennacl_2.10 --- > [INFO] Using 'UTF-8' encoding to copy filtered resources. > [INFO] skip non existing resourceDirectory /home/trevor/gits/mahout/ > viennacl/src/main/resources > [INFO] Copying 3 resources > [INFO] > [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile) @ > mahout-native-viennacl_2.10 --- > [INFO] Nothing to compile - all classes are up to date > [INFO] > [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ > mahout-native-viennacl_2.10 --- > [INFO] Nothing to compile - all classes are up to date > [INFO] > [INFO] --- exec-maven-plugin:1.2.1:exec (javacpp) @ > mahout-native-viennacl_2.10 --- > Warning: Could not load platform properties for class > org.apache.mahout.viennacl.opencl.GPUMMul > Warning: Could not load platform properties for class > org.apache.mahout.viennacl.opencl.GPUMMul$ > Generating /home/trevor/gits/mahout/viennacl/target/classes/org/ > apache/mahout/viennacl/opencl/javacpp/jniViennaCL.cpp > Compiling /home/trevor/gits/mahout/viennacl/target/classes/org/ > apache/mahout/viennacl/opencl/javacpp/linux-x86_64/libjniViennaCL.so > g++ -I/usr/include/viennacl -I/usr/lib/jvm/java-7-openjdk-amd64/include > -I/usr/lib/jvm/java-7-openjdk-amd64/include/linux > /home/trevor/gits/mahout/viennacl/target/classes/org/ > apache/mahout/viennacl/opencl/javacpp/jniViennaCL.cpp -msse3 -ffast-math > -fopenmp -fpermissive -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic > -march=native -m64 -Wall -Ofast -fPIC -shared -s -o libjniViennaCL.so > -lOpenCL > Deleting /home/trevor/gits/mahout/viennacl/target/classes/org/ > apache/mahout/viennacl/opencl/javacpp/jniViennaCL.cpp > [INFO] > [INFO] --- maven-resources-plugin:2.7:testResources > (default-testResources) @ mahout-native-viennacl_2.10 --- > [INFO] Using 'UTF-8' encoding to copy filtered resources. > [INFO] skip non existing resourceDirectory /home/trevor/gits/mahout/ > viennacl/src/test/resources > [INFO] Copying 3 resources > [INFO] > [INFO] --- scala-maven-plugin:3.2.0:testCompile (scala-test-compile) @ > mahout-native-viennacl_2.10 --- > [INFO] Nothing to compile - all classes are up to date > [INFO] > [INFO] --- maven-compiler-plugin:3.3:testCompile (default-testCompile) @ > mahout-native-viennacl_2.10 --- > [INFO] Nothing to compile - all classes are up to date > [INFO] > [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ > mahout-native-viennacl_2.10 --- > [INFO] Tests are skipped. > [INFO] > [INFO] --- scalatest-maven-plugin:1.0:test (test) @ > mahout-native-viennacl_2.10 --- > Discovery starting. > Discovery completed in 201 milliseconds. > Run starting. Expected test count is: 7 > ViennaCLSuiteVCL: > - row-major viennacl::matrix > + OCL matrix memory domain after assgn=2 > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > [INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul > solver > jvmRWRW > gpuRWCW > log4j:WARN No appenders could be found for logger > (org.apache.mahout.viennacl.opencl.GPUMMul$). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for > more info. > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > [INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul > solver > jvmRWRW > gpuRWCW > - dense vcl mmul with fast_copy > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > [INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul > solver > jvmRWRW > gpuRWCW > - mmul microbenchmark > + Mahout multiplication time: 2630 ms. > + ViennaCL/OpenCL multiplication time: 2072 ms. > + ViennaCL/cpu/OpenMP multiplication time: 813 ms. > - trans > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > [INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul > solver > gpuSparseRWRW > ViennaCL: FATAL ERROR: Kernel start failed for 'spgemm_stage3'. > ViennaCL: Smaller work sizes could not solve the problem. > ViennaCL: FATAL ERROR: CL_MEM_OBJECT_ALLOCATION_FAILURE > ViennaCL could not allocate memory on the device. Most likely the device > simply ran out of memory. > If you think that this is a bug in ViennaCL, please report it at > [email protected] and supply at least the following > information: > * Operating System > * Which OpenCL implementation (AMD, NVIDIA, etc.) > * ViennaCL version > Many thanks in advance!falling back to JVM MMUL > ViennaCL: FATAL ERROR: Kernel start failed for 'spgemm_stage3'. > ViennaCL: Smaller work sizes could not solve the problem. > - sparse mmul microbenchmark *** FAILED *** > java.lang.RuntimeException: ViennaCL: FATAL ERROR: > CL_MEM_OBJECT_ALLOCATION_FAILURE > ViennaCL could not allocate memory on the device. Most likely the device > simply ran out of memory. > If you think that this is a bug in ViennaCL, please report it at > [email protected] and supply at least the following > information: > * Operating System > * Which OpenCL implementation (AMD, NVIDIA, etc.) > * ViennaCL version > Many thanks in advance! > at > org.apache.mahout.viennacl.opencl.javacpp.CompressedMatrix.allocate(Native > Method) > at org.apache.mahout.viennacl.opencl.javacpp.CompressedMatrix.<init>( > CompressedMatrix.scala:61) > at org.apache.mahout.viennacl.opencl.ViennaCLSuiteVCL$$ > anonfun$5.apply$mcV$sp(ViennaCLSuiteVCL.scala:250) > at org.apache.mahout.viennacl.opencl.ViennaCLSuiteVCL$$anonfun$5.apply( > ViennaCLSuiteVCL.scala:218) > at org.apache.mahout.viennacl.opencl.ViennaCLSuiteVCL$$anonfun$5.apply( > ViennaCLSuiteVCL.scala:218) > at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp( > Transformer.scala:22) > at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) > at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) > at org.scalatest.Transformer.apply(Transformer.scala:22) > at org.scalatest.Transformer.apply(Transformer.scala:20) > ... > + Mahout Sparse multiplication time: 13303 ms. > - VCL Dense Matrix %*% Dense vector > + Mahout dense matrix %*% dense vector multiplication time: 0 ms. > + ViennaCL/cpu/OpenMP dense matrix %*% dense vector multiplication time: > 4 ms. > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > [INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul > solver > gpuRWCW > - Sparse %*% Dense mmul microbenchmark > + Mahout multiplication time: 821 ms. > + ViennaCL/OpenCL multiplication time: 754 ms. > + ViennaCL/cpu/OpenMP multiplication time: 1473 ms. > Run completed in 26 seconds, 944 milliseconds. > Total number of tests run: 7 > Suites: completed 2, aborted 0 > Tests: succeeded 6, failed 1, canceled 0, ignored 0, pending 0 > *** 1 TEST FAILED *** > [INFO] ------------------------------------------------------------ > ------------ > [INFO] BUILD FAILURE > [INFO] ------------------------------------------------------------ > ------------ > [INFO] Total time: 45.600 s > [INFO] Finished at: 2016-10-28T17:58:45-05:00 > [INFO] Final Memory: 20M/309M > [INFO] ------------------------------------------------------------ > ------------ > [ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0:test > (test) on project mahout-native-viennacl_2.10: There are test failures -> > [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, > please read the following articles: > [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ > MojoFailureException > > > Trevor Grant > Data Scientist > https://github.com/rawkintrevo > http://stackexchange.com/users/3002022/rawkintrevo > http://trevorgrant.org > > *"Fortunate is he, who is able to know the causes of things." -Virgil* > > > On Fri, Oct 28, 2016 at 5:38 PM, Andrew Musselman < > [email protected]> wrote: > >> > >> > My Desktop >> >> > >> Branch: master >> >> > Build Command: mvn clean install -Phadoop2 >> >> > Built and tested successfully >> >> > >> Branch: andrewpalumbo/mahout/viennacl-opmmul-a >> >> > Bulid Command: mvn clean install -Pviennacl -Phadoop2 >> >> Builds successfully- >> >> > Tests still riddled with >> >> > >> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver >> >> > [WARN] Unable to create class GPUMMul: attempting OpenMP version >> >> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver >> >> > [INFO] Unable to create class OMPMMul: falling back to java version >> >> > >> Also OOM >> ViennaCLSuiteOMP: >> - row-major viennacl::matrix >> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver >> [WARN] Unable to create class GPUMMul: attempting OpenMP version >> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver >> [INFO] Unable to create class OMPMMul: falling back to java version >> - mmul microbenchmark >> + Mahout multiplication time: 8875 ms. >> + ViennaCL/cpu/OpenMP multiplication time: 599 ms. >> - trans >> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver >> [WARN] Unable to create class GPUMMul: attempting OpenMP version >> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver >> [INFO] Unable to create class OMPMMul: falling back to java version >> *** RUN ABORTED *** >> java.lang.OutOfMemoryError: Java heap space >> at >> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.rehash(Int2 >> DoubleOpenHashMap.java:1059) >> at >> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.insert(Int2 >> DoubleOpenHashMap.java:295) >> at >> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.put(Int2Dou >> bleOpenHashMap.java:301) >> at >> org.apache.mahout.math.RandomAccessSparseVector.setQuick(Ran >> domAccessSparseVector.java:130) >> at >> org.apache.mahout.math.SparseRowMatrix.setQuick(SparseRowMatrix.java:105) >> at org.apache.mahout.math.AbstractMatrix.assign(AbstractMatrix. >> java:256) >> at >> org.apache.mahout.math.scalabindings.MatrixOps.$colon$eq( >> MatrixOps.scala:192) >> at >> org.apache.mahout.math.scalabindings.MatrixOps.cloned( >> MatrixOps.scala:260) >> at >> org.apache.mahout.math.scalabindings.MatrixOps.$minus(MatrixOps.scala:66) >> at >> org.apache.mahout.viennacl.openmp.ViennaCLSuiteOMP$$anonfun$ >> 4.apply$mcV$sp(ViennaCLSuiteOMP.scala:152) >> >> > >> >> Machine Info: >> >> > ============================================================ >> ==================== >> >> > >> os major/minor: >> >> > $ lsb_release -a >> No LSB modules are available. >> Distributor ID: Ubuntu >> Description: Ubuntu 16.04.1 LTS >> Release: 16.04 >> Codename: xenial >> >> > >> >> chip arch: >> >> > $ uname -a >> Linux Bob 4.4.0-34-generic #53-Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 >> x86_64 x86_64 x86_64 GNU/Linux >> >> > >> $ cat /proc/cpuinfo >> >> > ... >> >> > model name : Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz >> >> > ... >> >> > ^^ 6 cores >> >> > >> >> GPU: >> >> > $ sudo nvidia-smi >> >> > >> +----------------------------------------------------------- >> ------------------+ >> | NVIDIA-SMI 367.44 Driver Version: 367.44 >> | >> |-------------------------------+----------------------+---- >> ------------------+ >> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. >> ECC | >> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute >> M. | >> |===============================+======================+==== >> ==================| >> | 0 GeForce GTX 750 Ti Off | 0000:01:00.0 On | >> N/A | >> | 29% 25C P8 1W / 38W | 307MiB / 1998MiB | 0% >> Default | >> +-------------------------------+----------------------+---- >> ------------------+ >> >> >> +----------------------------------------------------------- >> ------------------+ >> | Processes: GPU >> Memory | >> | GPU PID Type Process name Usage >> | >> |=========================================================== >> ==================| >> | 0 1537 G ...C-EnableWebRtcEcdsa,WebRTC-H264WithOpenH2 >> 87MiB | >> | 0 1990 G /usr/lib/xorg/Xorg >> 101MiB | >> | 0 2319 G /usr/bin/gnome-shell >> 104MiB | >> | 0 4360 G /usr/lib/xorg/Xorg >> 12MiB | >> +----------------------------------------------------------- >> ------------------+ >> >> > >> >> >> >> clinfo output: >> >> > $ clinfo >> Number of platforms 1 >> Platform Name NVIDIA CUDA >> Platform Vendor NVIDIA Corporation >> Platform Version OpenCL 1.2 CUDA 8.0.0 >> Platform Profile FULL_PROFILE >> Platform Extensions >> cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics >> cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics >> cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing >> cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll >> cl_nv_copy_opts >> Platform Extensions function suffix NV >> >> Platform Name NVIDIA CUDA >> Number of devices 1 >> Device Name GeForce GTX 750 Ti >> Device Vendor NVIDIA Corporation >> Device Vendor ID 0x10de >> Device Version OpenCL 1.2 CUDA >> Driver Version 367.44 >> Device OpenCL C Version OpenCL C 1.2 >> Device Type GPU >> Device Profile FULL_PROFILE >> Device Topology (NV) PCI-E, 01:00.0 >> Max compute units 5 >> Max clock frequency 1150MHz >> Compute Capability (NV) 5.0 >> Device Partition (core) >> Max number of sub-devices 1 >> Supported partition types None >> Max work item dimensions 3 >> Max work item sizes 1024x1024x64 >> Max work group size 1024 >> Preferred work group size multiple 32 >> Warp size (NV) 32 >> Preferred / native vector sizes >> char 1 / 1 >> short 1 / 1 >> int 1 / 1 >> long 1 / 1 >> half 0 / 0 >> (n/a) >> float 1 / 1 >> double 1 / 1 >> (cl_khr_fp64) >> Half-precision Floating-point support (n/a) >> Single-precision Floating-point support (core) >> Denormals Yes >> Infinity and NANs Yes >> Round to nearest Yes >> Round to zero Yes >> Round to infinity Yes >> IEEE754-2008 fused multiply-add Yes >> Support is emulated in software No >> Correctly-rounded divide and sqrt operations Yes >> Double-precision Floating-point support (cl_khr_fp64) >> Denormals Yes >> Infinity and NANs Yes >> Round to nearest Yes >> Round to zero Yes >> Round to infinity Yes >> IEEE754-2008 fused multiply-add Yes >> Support is emulated in software No >> Correctly-rounded divide and sqrt operations No >> Address bits 64, Little-Endian >> Global memory size 2095841280 (1.952GiB) >> Error Correction support No >> Max memory allocation 523960320 (499.7MiB) >> Unified memory for Host and Device No >> Integrated memory (NV) No >> Minimum alignment for any data type 128 bytes >> Alignment of base address 4096 bits (512 bytes) >> Global Memory cache type Read/Write >> Global Memory cache size 81920 >> Global Memory cache line 128 bytes >> Image support Yes >> Max number of samplers per kernel 32 >> Max size for 1D images from buffer 134217728 pixels >> Max 1D or 2D image array size 2048 images >> Max 2D image size 16384x16384 pixels >> Max 3D image size 4096x4096x4096 pixels >> Max number of read image args 256 >> Max number of write image args 16 >> Local memory type Local >> Local memory size 49152 (48KiB) >> Registers per block (NV) 65536 >> Max constant buffer size 65536 (64KiB) >> Max number of constant args 9 >> Max size of kernel argument 4352 (4.25KiB) >> Queue properties >> Out-of-order execution Yes >> Profiling Yes >> Prefer user sync for interop No >> Profiling timer resolution 1000ns >> Execution capabilities >> Run OpenCL kernels Yes >> Run native kernels No >> Kernel execution timeout (NV) Yes >> Concurrent copy and kernel execution (NV) Yes >> Number of async copy engines 1 >> printf() buffer size 1048576 (1024KiB) >> Built-in kernels >> Device Available Yes >> Compiler Available Yes >> Linker Available Yes >> Device Extensions >> cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics >> cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics >> cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing >> cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll >> cl_nv_copy_opts >> >> NULL platform behavior >> clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) NVIDIA CUDA >> clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) Success [NV] >> clCreateContext(NULL, ...) [default] Success [NV] >> clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in >> platform >> clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) No platform >> clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices >> found in platform >> clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) No devices found >> in >> platform >> clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) No platform >> >> ICD loader properties >> ICD loader Name OpenCL ICD Loader >> ICD loader Vendor OCL Icd free software >> ICD loader Version 2.2.8 >> ICD loader Profile OpenCL 1.2 >> NOTE: your OpenCL library declares to support OpenCL 1.2, >> but it seems to support up to OpenCL 2.1 too. >> >> > >> >> Java major/minor and vendor: >> >> > $ java -version >> openjdk version "1.8.0_91" >> OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-3ubuntu1~16. >> 04.1-b14) >> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) >> >> > >> >> vcl version, >> >> > (Ignoring what is in /usr/include/viennacl/version.hpp) >> >> > 1.7.1 >> >> >> gcc version: >> >> > $ gcc --version >> gcc (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609 >> >> > >> >> On Wed, Oct 26, 2016 at 11:31 PM, Andrew Palumbo <[email protected]> >> wrote: >> >> > >> > >> > >> > Could everybody please post any errors when stack trace that they are >> > getting on master or the vcl pr branch along with full system info? >> > >> > Ie. Branch, os major/minor, chip arch, GPU, clinfo output, Java >> > major/minor and vendor, vcl version, gcc version, and anything else >> that >> > may be useful so that we may compare? >> > >> > Thx >> > >> > Andy >> > >> > >> > Sent from my Galaxy Tab A >> > >> > >
