Ok this makes sense.. it's not actually a (java) OOM error, it's actually a blanket GPU error... that's why we have the fall backs to omp and jvm..
Does it happen each time you run that test? Some times these are one off things where the gpu buffer had not been cleared when the next test starts.. or something like that. Thanks this is a big help in figuring out what's happening. Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Trevor Grant <[email protected]> Date: 10/28/2016 4:01 PM (GMT-08:00) To: [email protected] Subject: Re: Errors when 0.13.0 master or vcl I was finally able to hit an OOM error. I initially was running everything from the top. When I run mvn test from the viennacl directory... /mahout/viennacl$ mvn test [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.mahout:mahout-native-viennacl_2.10:jar:0.13.0-SNAPSHOT [WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.mahout:mahout-native-viennacl_${scala.compat.version}:[unknown-version], /home/trevor/gits/mahout/viennacl/pom.xml, line 31, column 15 [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin org.codehaus.mojo:exec-maven-plugin @ org.apache.mahout:mahout-native-viennacl_${scala.compat.version}:[unknown-version], /home/trevor/gits/mahout/viennacl/pom.xml, line 181, column 15 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Mahout Native VienniaCL OpenCL Bindings 0.13.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-enforcer-plugin:1.4:enforce (enforce-versions) @ mahout-native-viennacl_2.10 --- [INFO] [INFO] --- scala-maven-plugin:3.2.0:add-source (add-scala-sources) @ mahout-native-viennacl_2.10 --- [INFO] Add Source directory: /home/trevor/gits/mahout/viennacl/src/main/scala [INFO] Add Test Source directory: /home/trevor/gits/mahout/viennacl/src/test/scala [INFO] [INFO] --- maven-dependency-plugin:2.3:properties (default) @ mahout-native-viennacl_2.10 --- [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ mahout-native-viennacl_2.10 --- [INFO] [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ mahout-native-viennacl_2.10 --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/trevor/gits/mahout/viennacl/src/main/resources [INFO] Copying 3 resources [INFO] [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile) @ mahout-native-viennacl_2.10 --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ mahout-native-viennacl_2.10 --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- exec-maven-plugin:1.2.1:exec (javacpp) @ mahout-native-viennacl_2.10 --- Warning: Could not load platform properties for class org.apache.mahout.viennacl.opencl.GPUMMul Warning: Could not load platform properties for class org.apache.mahout.viennacl.opencl.GPUMMul$ Generating /home/trevor/gits/mahout/viennacl/target/classes/org/apache/mahout/viennacl/opencl/javacpp/jniViennaCL.cpp Compiling /home/trevor/gits/mahout/viennacl/target/classes/org/apache/mahout/viennacl/opencl/javacpp/linux-x86_64/libjniViennaCL.so g++ -I/usr/include/viennacl -I/usr/lib/jvm/java-7-openjdk-amd64/include -I/usr/lib/jvm/java-7-openjdk-amd64/include/linux /home/trevor/gits/mahout/viennacl/target/classes/org/apache/mahout/viennacl/opencl/javacpp/jniViennaCL.cpp -msse3 -ffast-math -fopenmp -fpermissive -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -march=native -m64 -Wall -Ofast -fPIC -shared -s -o libjniViennaCL.so -lOpenCL Deleting /home/trevor/gits/mahout/viennacl/target/classes/org/apache/mahout/viennacl/opencl/javacpp/jniViennaCL.cpp [INFO] [INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ mahout-native-viennacl_2.10 --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/trevor/gits/mahout/viennacl/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- scala-maven-plugin:3.2.0:testCompile (scala-test-compile) @ mahout-native-viennacl_2.10 --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-compiler-plugin:3.3:testCompile (default-testCompile) @ mahout-native-viennacl_2.10 --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ mahout-native-viennacl_2.10 --- [INFO] Tests are skipped. [INFO] [INFO] --- scalatest-maven-plugin:1.0:test (test) @ mahout-native-viennacl_2.10 --- Discovery starting. Discovery completed in 201 milliseconds. Run starting. Expected test count is: 7 ViennaCLSuiteVCL: - row-major viennacl::matrix + OCL matrix memory domain after assgn=2 [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver [INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver jvmRWRW gpuRWCW log4j:WARN No appenders could be found for logger (org.apache.mahout.viennacl.opencl.GPUMMul$). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver [INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver jvmRWRW gpuRWCW - dense vcl mmul with fast_copy [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver [INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver jvmRWRW gpuRWCW - mmul microbenchmark + Mahout multiplication time: 2630 ms. + ViennaCL/OpenCL multiplication time: 2072 ms. + ViennaCL/cpu/OpenMP multiplication time: 813 ms. - trans [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver [INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver gpuSparseRWRW ViennaCL: FATAL ERROR: Kernel start failed for 'spgemm_stage3'. ViennaCL: Smaller work sizes could not solve the problem. ViennaCL: FATAL ERROR: CL_MEM_OBJECT_ALLOCATION_FAILURE ViennaCL could not allocate memory on the device. Most likely the device simply ran out of memory. If you think that this is a bug in ViennaCL, please report it at [email protected] and supply at least the following information: * Operating System * Which OpenCL implementation (AMD, NVIDIA, etc.) * ViennaCL version Many thanks in advance!falling back to JVM MMUL ViennaCL: FATAL ERROR: Kernel start failed for 'spgemm_stage3'. ViennaCL: Smaller work sizes could not solve the problem. - sparse mmul microbenchmark *** FAILED *** java.lang.RuntimeException: ViennaCL: FATAL ERROR: CL_MEM_OBJECT_ALLOCATION_FAILURE ViennaCL could not allocate memory on the device. Most likely the device simply ran out of memory. If you think that this is a bug in ViennaCL, please report it at [email protected] and supply at least the following information: * Operating System * Which OpenCL implementation (AMD, NVIDIA, etc.) * ViennaCL version Many thanks in advance! at org.apache.mahout.viennacl.opencl.javacpp.CompressedMatrix.allocate(Native Method) at org.apache.mahout.viennacl.opencl.javacpp.CompressedMatrix.<init>(CompressedMatrix.scala:61) at org.apache.mahout.viennacl.opencl.ViennaCLSuiteVCL$$anonfun$5.apply$mcV$sp(ViennaCLSuiteVCL.scala:250) at org.apache.mahout.viennacl.opencl.ViennaCLSuiteVCL$$anonfun$5.apply(ViennaCLSuiteVCL.scala:218) at org.apache.mahout.viennacl.opencl.ViennaCLSuiteVCL$$anonfun$5.apply(ViennaCLSuiteVCL.scala:218) at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22) at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) ... + Mahout Sparse multiplication time: 13303 ms. - VCL Dense Matrix %*% Dense vector + Mahout dense matrix %*% dense vector multiplication time: 0 ms. + ViennaCL/cpu/OpenMP dense matrix %*% dense vector multiplication time: 4 ms. [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver [INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver gpuRWCW - Sparse %*% Dense mmul microbenchmark + Mahout multiplication time: 821 ms. + ViennaCL/OpenCL multiplication time: 754 ms. + ViennaCL/cpu/OpenMP multiplication time: 1473 ms. Run completed in 26 seconds, 944 milliseconds. Total number of tests run: 7 Suites: completed 2, aborted 0 Tests: succeeded 6, failed 1, canceled 0, ignored 0, pending 0 *** 1 TEST FAILED *** [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 45.600 s [INFO] Finished at: 2016-10-28T17:58:45-05:00 [INFO] Final Memory: 20M/309M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0:test (test) on project mahout-native-viennacl_2.10: There are test failures -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException Trevor Grant Data Scientist https://github.com/rawkintrevo http://stackexchange.com/users/3002022/rawkintrevo http://trevorgrant.org *"Fortunate is he, who is able to know the causes of things." -Virgil* On Fri, Oct 28, 2016 at 5:38 PM, Andrew Musselman < [email protected]> wrote: > > > > My Desktop > > > > Branch: master > > > Build Command: mvn clean install -Phadoop2 > > > Built and tested successfully > > > > Branch: andrewpalumbo/mahout/viennacl-opmmul-a > > > Bulid Command: mvn clean install -Pviennacl -Phadoop2 > > Builds successfully- > > > Tests still riddled with > > > > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > > > [WARN] Unable to create class GPUMMul: attempting OpenMP version > > > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver > > > [INFO] Unable to create class OMPMMul: falling back to java version > > > > Also OOM > ViennaCLSuiteOMP: > - row-major viennacl::matrix > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > [WARN] Unable to create class GPUMMul: attempting OpenMP version > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver > [INFO] Unable to create class OMPMMul: falling back to java version > - mmul microbenchmark > + Mahout multiplication time: 8875 ms. > + ViennaCL/cpu/OpenMP multiplication time: 599 ms. > - trans > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > [WARN] Unable to create class GPUMMul: attempting OpenMP version > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver > [INFO] Unable to create class OMPMMul: falling back to java version > *** RUN ABORTED *** > java.lang.OutOfMemoryError: Java heap space > at > it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.rehash( > Int2DoubleOpenHashMap.java:1059) > at > it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.insert( > Int2DoubleOpenHashMap.java:295) > at > it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.put( > Int2DoubleOpenHashMap.java:301) > at > org.apache.mahout.math.RandomAccessSparseVector.setQuick( > RandomAccessSparseVector.java:130) > at > org.apache.mahout.math.SparseRowMatrix.setQuick(SparseRowMatrix.java:105) > at org.apache.mahout.math.AbstractMatrix.assign(AbstractMatrix.java:256) > at > org.apache.mahout.math.scalabindings.MatrixOps.$ > colon$eq(MatrixOps.scala:192) > at > org.apache.mahout.math.scalabindings.MatrixOps.cloned(MatrixOps.scala:260) > at > org.apache.mahout.math.scalabindings.MatrixOps.$minus(MatrixOps.scala:66) > at > org.apache.mahout.viennacl.openmp.ViennaCLSuiteOMP$$ > anonfun$4.apply$mcV$sp(ViennaCLSuiteOMP.scala:152) > > > > > Machine Info: > > > ============================================================ > ==================== > > > > os major/minor: > > > $ lsb_release -a > No LSB modules are available. > Distributor ID: Ubuntu > Description: Ubuntu 16.04.1 LTS > Release: 16.04 > Codename: xenial > > > > > chip arch: > > > $ uname -a > Linux Bob 4.4.0-34-generic #53-Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 > x86_64 x86_64 x86_64 GNU/Linux > > > > $ cat /proc/cpuinfo > > > ... > > > model name : Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz > > > ... > > > ^^ 6 cores > > > > > GPU: > > > $ sudo nvidia-smi > > > > +----------------------------------------------------------- > ------------------+ > | NVIDIA-SMI 367.44 Driver Version: 367.44 > | > |-------------------------------+----------------------+---- > ------------------+ > | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. > ECC | > | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute > M. | > |===============================+======================+==== > ==================| > | 0 GeForce GTX 750 Ti Off | 0000:01:00.0 On | > N/A | > | 29% 25C P8 1W / 38W | 307MiB / 1998MiB | 0% > Default | > +-------------------------------+----------------------+---- > ------------------+ > > > +----------------------------------------------------------- > ------------------+ > | Processes: GPU > Memory | > | GPU PID Type Process name Usage > | > |=========================================================== > ==================| > | 0 1537 G ...C-EnableWebRtcEcdsa,WebRTC-H264WithOpenH2 > 87MiB | > | 0 1990 G /usr/lib/xorg/Xorg > 101MiB | > | 0 2319 G /usr/bin/gnome-shell > 104MiB | > | 0 4360 G /usr/lib/xorg/Xorg > 12MiB | > +----------------------------------------------------------- > ------------------+ > > > > > > > clinfo output: > > > $ clinfo > Number of platforms 1 > Platform Name NVIDIA CUDA > Platform Vendor NVIDIA Corporation > Platform Version OpenCL 1.2 CUDA 8.0.0 > Platform Profile FULL_PROFILE > Platform Extensions > cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics > cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics > cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing > cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll > cl_nv_copy_opts > Platform Extensions function suffix NV > > Platform Name NVIDIA CUDA > Number of devices 1 > Device Name GeForce GTX 750 Ti > Device Vendor NVIDIA Corporation > Device Vendor ID 0x10de > Device Version OpenCL 1.2 CUDA > Driver Version 367.44 > Device OpenCL C Version OpenCL C 1.2 > Device Type GPU > Device Profile FULL_PROFILE > Device Topology (NV) PCI-E, 01:00.0 > Max compute units 5 > Max clock frequency 1150MHz > Compute Capability (NV) 5.0 > Device Partition (core) > Max number of sub-devices 1 > Supported partition types None > Max work item dimensions 3 > Max work item sizes 1024x1024x64 > Max work group size 1024 > Preferred work group size multiple 32 > Warp size (NV) 32 > Preferred / native vector sizes > char 1 / 1 > short 1 / 1 > int 1 / 1 > long 1 / 1 > half 0 / 0 (n/a) > float 1 / 1 > double 1 / 1 > (cl_khr_fp64) > Half-precision Floating-point support (n/a) > Single-precision Floating-point support (core) > Denormals Yes > Infinity and NANs Yes > Round to nearest Yes > Round to zero Yes > Round to infinity Yes > IEEE754-2008 fused multiply-add Yes > Support is emulated in software No > Correctly-rounded divide and sqrt operations Yes > Double-precision Floating-point support (cl_khr_fp64) > Denormals Yes > Infinity and NANs Yes > Round to nearest Yes > Round to zero Yes > Round to infinity Yes > IEEE754-2008 fused multiply-add Yes > Support is emulated in software No > Correctly-rounded divide and sqrt operations No > Address bits 64, Little-Endian > Global memory size 2095841280 (1.952GiB) > Error Correction support No > Max memory allocation 523960320 (499.7MiB) > Unified memory for Host and Device No > Integrated memory (NV) No > Minimum alignment for any data type 128 bytes > Alignment of base address 4096 bits (512 bytes) > Global Memory cache type Read/Write > Global Memory cache size 81920 > Global Memory cache line 128 bytes > Image support Yes > Max number of samplers per kernel 32 > Max size for 1D images from buffer 134217728 pixels > Max 1D or 2D image array size 2048 images > Max 2D image size 16384x16384 pixels > Max 3D image size 4096x4096x4096 pixels > Max number of read image args 256 > Max number of write image args 16 > Local memory type Local > Local memory size 49152 (48KiB) > Registers per block (NV) 65536 > Max constant buffer size 65536 (64KiB) > Max number of constant args 9 > Max size of kernel argument 4352 (4.25KiB) > Queue properties > Out-of-order execution Yes > Profiling Yes > Prefer user sync for interop No > Profiling timer resolution 1000ns > Execution capabilities > Run OpenCL kernels Yes > Run native kernels No > Kernel execution timeout (NV) Yes > Concurrent copy and kernel execution (NV) Yes > Number of async copy engines 1 > printf() buffer size 1048576 (1024KiB) > Built-in kernels > Device Available Yes > Compiler Available Yes > Linker Available Yes > Device Extensions > cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics > cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics > cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing > cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll > cl_nv_copy_opts > > NULL platform behavior > clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) NVIDIA CUDA > clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) Success [NV] > clCreateContext(NULL, ...) [default] Success [NV] > clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in > platform > clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) No platform > clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices > found in platform > clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) No devices found in > platform > clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) No platform > > ICD loader properties > ICD loader Name OpenCL ICD Loader > ICD loader Vendor OCL Icd free software > ICD loader Version 2.2.8 > ICD loader Profile OpenCL 1.2 > NOTE: your OpenCL library declares to support OpenCL 1.2, > but it seems to support up to OpenCL 2.1 too. > > > > > Java major/minor and vendor: > > > $ java -version > openjdk version "1.8.0_91" > OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14) > OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) > > > > > vcl version, > > > (Ignoring what is in /usr/include/viennacl/version.hpp) > > > 1.7.1 > > > gcc version: > > > $ gcc --version > gcc (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609 > > > > > On Wed, Oct 26, 2016 at 11:31 PM, Andrew Palumbo <[email protected]> > wrote: > > > > > > > > > Could everybody please post any errors when stack trace that they are > > getting on master or the vcl pr branch along with full system info? > > > > Ie. Branch, os major/minor, chip arch, GPU, clinfo output, Java > > major/minor and vendor, vcl version, gcc version, and anything else that > > may be useful so that we may compare? > > > > Thx > > > > Andy > > > > > > Sent from my Galaxy Tab A > > >
