I think that the memory should be set to 3 or 4G in the tests in pom.xml.. maybe you're o erring that setting?
Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: Andrew Musselman <andrew.mussel...@gmail.com> Date: 10/17/2016 9:47 AM (GMT-08:00) To: dev@mahout.apache.org Subject: Re: Trying out PR 261 Using the viennacl-omp profile I'm getting an out of memory error even with Xmx set in MAHOUT_OPTS to 1g: - mmul microbenchmark + Mahout multiplication time: 8920 ms. + ViennaCL/cpu/OpenMP multiplication time: 616 ms. - trans [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver [WARN] Unable to create class GPUMMul: attempting OpenMP version [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver [INFO] Unable to create class OMPMMul: falling back to java version *** RUN ABORTED *** java.lang.OutOfMemoryError: Java heap space at it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.rehash(Int2DoubleOpenHashMap.java:1059) at it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.insert(Int2DoubleOpenHashMap.java:295) at it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.put(Int2DoubleOpenHashMap.java:301) at org.apache.mahout.math.RandomAccessSparseVector.setQuick(RandomAccessSparseVector.java:130) at org.apache.mahout.math.SparseRowMatrix.setQuick(SparseRowMatrix.java:105) at org.apache.mahout.math.AbstractMatrix.assign(AbstractMatrix.java:256) at org.apache.mahout.math.scalabindings.MatrixOps.$colon$eq(MatrixOps.scala:192) at org.apache.mahout.math.scalabindings.MatrixOps.cloned(MatrixOps.scala:260) at org.apache.mahout.math.scalabindings.MatrixOps.$minus(MatrixOps.scala:66) at org.apache.mahout.viennacl.openmp.ViennaCLSuiteOMP$$anonfun$4.apply$mcV$sp(ViennaCLSuiteOMP.scala:152) ... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Mahout Build Tools ................................. SUCCESS [ 1.131 s] [INFO] Apache Mahout ...................................... SUCCESS [ 0.023 s] [INFO] Mahout Math ........................................ SUCCESS [ 31.439 s] [INFO] Mahout HDFS ........................................ SUCCESS [ 2.151 s] [INFO] Mahout Math Scala bindings ......................... SUCCESS [02:30 min] [INFO] Mahout Spark bindings .............................. SUCCESS [ 43.884 s] [INFO] Mahout Flink bindings .............................. SUCCESS [04:13 min] [INFO] Mahout Spark bindings shell ........................ SUCCESS [ 4.071 s] [INFO] Mahout Release Package ............................. SUCCESS [ 1.311 s] [INFO] Mahout H2O backend ................................. SUCCESS [02:50 min] [INFO] Mahout Native VienniaCL OpenMP Bindings ............ FAILURE [ 36.106 s] On Sun, Oct 16, 2016 at 9:43 PM, Trevor Grant <trevor.d.gr...@gmail.com> wrote: > NOTE: > > The last solution creates a cyclical dependency- > > The way I made it work was happen-stance. > > 1) > mvn clean install -Pviennacl -Phadoop2 -DskipTests > > 2) > update the pom as described previously > > 3) > cd math-scala > > 4) > mvn test > > that will make it work- hacky but POC. > > Next how to do this 'right'. > > tg > > > > Trevor Grant > Data Scientist > https://github.com/rawkintrevo > http://stackexchange.com/users/3002022/rawkintrevo > http://trevorgrant.org > > *"Fortunate is he, who is able to know the causes of things." -Virgil* > > > On Sun, Oct 16, 2016 at 11:29 PM, Trevor Grant <trevor.d.gr...@gmail.com> > wrote: > > > Maybe not the long term fix, but adding > > > > <dependency> > > <groupId>org.apache.mahout</groupId> > > <artifactId>mahout-native-viennacl_${scala.compat. > > version}</artifactId> > > <scope>test</scope> > > </dependency> > > > > to math-scala/pom.xml seems to fix the afore mentioned errors > > > > > > Trevor Grant > > Data Scientist > > https://github.com/rawkintrevo > > http://stackexchange.com/users/3002022/rawkintrevo > > http://trevorgrant.org > > > > *"Fortunate is he, who is able to know the causes of things." -Virgil* > > > > > > On Sun, Oct 16, 2016 at 10:23 PM, Andrew Palumbo <ap....@outlook.com> > > wrote: > > > >> Thanks Trevor and Andrew- I think that both of your environments are > >> fine.. (Ubuntu 16.04 was just to recommended because ViennaCL 1.7.1 is > >> easily installed with sudo-apt-get install. As logn as you have 1.7.1 > on > >> the correct path, you should be fine. > >> > >> > >> I'm getting these errors now also, and by viewing the stack trace, I can > >> see that the mahout_native_vienniacl*.jars did not make it onto the > >> classpath some how: > >> > >> > >> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > >> [WARN] Unable to create class GPUMMul: attempting OpenMP version > >> java.lang.ClassNotFoundException: org.apache.mahout.viennacl.ope > >> ncl.GPUMMul$ > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > >> at java.lang.Class.forName0(Native Method) > >> at java.lang.Class.forName(Class.java:264) > >> at org.apache.mahout.math.scalabindings.SolverFactory$.getOpera > >> tor(SolverFactory.scala:50) > >> > >> > >> I think that i might have merged some conflicts badly when I rebased > last > >> week, because I was not seeing these errors. I'll tey to get this > worked > >> out in the next couple of days. > >> > >> > >> > >> ________________________________ > >> From: Trevor Grant <trevor.d.gr...@gmail.com> > >> Sent: Sunday, October 16, 2016 11:04:46 PM > >> To: dev@mahout.apache.org > >> Subject: Re: Trying out PR 261 > >> > >> Having similar issues- > >> > >> Running 14.04 (I know, supposed to be 16.04, but whatever bear with me) > >> > >> Downloaded viennacl source, unzipped tar-dir/viennacl to > >> /usr/include/viennacl (e.g. 'installed' viennacl 1.7.1) > >> > >> Updated gcc (though I don't think this was neccessary, now that I see > >> Andrew M was having the same issues) > >> sudo add-apt-repository ppa:ubuntu-toolchain-r/test > >> sudo apt-get update > >> sudo apt-get install gcc-5 g++-5 > >> > >> sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 0 > >> --slave /usr/bin/g++ g++ /usr/bin/g++-5 > >> > >> > >> System / GPU info > >> +------------------------------------------------------+ > >> > >> | NVIDIA-SMI 352.63 Driver Version: 352.63 | > >> > >> |-------------------------------+----------------------+---- > >> ------------------+ > >> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile > Uncorr. > >> ECC | > >> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util > Compute > >> M. | > >> |===============================+======================+==== > >> ==================| > >> | 0 GeForce GT 740 Off | 0000:02:00.0 N/A | > >> N/A | > >> | 33% 34C P8 N/A / N/A | 411MiB / 1021MiB | N/A > >> Default | > >> +-------------------------------+----------------------+---- > >> ------------------+ > >> > >> > >> > >> > >> Build was successful- no problems there. > >> > >> Still getting a slew of: > >> > >> [32mMatlabLikeMatrixOpsSuite: [0m > >> [32m- multiplication [0m > >> [32m- Hadamard [0m > >> [32mRLikeVectorOpsSuite: [0m > >> [32m- Hadamard [0m > >> 19489 [ScalaTest-main-running-RLikeVectorOpsSuite] DEBUG > >> org.apache.mahout.math.scalabindings.RLikeVectorOpsSuite - dense > vector > >> dots:127.00 ms. > >> 19489 [ScalaTest-main-running-RLikeVectorOpsSuite] DEBUG > >> org.apache.mahout.math.scalabindings.RLikeVectorOpsSuite - dense view > >> dots:201.00 ms. > >> [32m- dot-view performance [0m > >> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > >> [WARN] Unable to create class GPUMMul: attempting OpenMP version > >> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver > >> [INFO] Unable to create class OMPMMul: falling back to java version > >> 19491 [ScalaTest-main-running-MathSuite] DEBUG > >> org.apache.mahout.math.scalabindings.MMul$ - AAt computation detected. > >> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > >> [WARN] Unable to create class GPUMMul: attempting OpenMP version > >> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver > >> [INFO] Unable to create class OMPMMul: falling back to java version > >> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > >> [WARN] Unable to create class GPUMMul: attempting OpenMP version > >> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver > >> [INFO] Unable to create class OMPMMul: falling back to java version > >> ... goes on like that for a long long time > >> > >> > >> > >> Trevor Grant > >> Data Scientist > >> https://github.com/rawkintrevo > >> http://stackexchange.com/users/3002022/rawkintrevo > >> http://trevorgrant.org > >> > >> *"Fortunate is he, who is able to know the causes of things." -Virgil* > >> > >> > >> On Sun, Oct 16, 2016 at 6:59 PM, Andrew Musselman < > >> andrew.mussel...@gmail.com> wrote: > >> > >> > I'm not sure my setup is correct; on Ubuntu 16.04 I'm getting a > >> firehose of > >> > infos and warns about not finding the GPUMMul and OMPMMul classes. > I've > >> > installed libviennacl-dev 1.7.1 through apt-get. Any ideas? > >> > > >> > Here's repro: > >> > $ git fetch origin pull/261/head:viennacl-opmmul-a > >> > $ mvn clean install -Pviennacl -Phadoop2 > >> > > >> > Infos and warns (these are littered consistently throughout test > logs): > >> > %*% B mapBlock {} > >> > - C = A %*% B incompatible B keys > >> > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > >> > [WARN] Unable to create class GPUMMul: attempting OpenMP version > >> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver > >> > [INFO] Unable to create class OMPMMul: falling back to java version > >> > - Spark-specific C = At %*% B , join > >> > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > >> > [WARN] Unable to create class GPUMMul: attempting OpenMP version > >> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver > >> > [INFO] Unable to create class OMPMMul: falling back to java version > >> > - C = At %*% B , join, String-keyed > >> > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > >> > [WARN] Unable to create class GPUMMul: attempting OpenMP version > >> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver > >> > [INFO] Unable to create class OMPMMul: falling back to java version > >> > - C = At %*% B , zippable, String-keyed > >> > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > >> > [WARN] Unable to create class GPUMMul: attempting OpenMP version > >> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver > >> > [INFO] Unable to create class OMPMMul: falling back to java version > >> > - C = A %*% B.t > >> > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > >> > [WARN] Unable to create class GPUMMul: attempting OpenMP version > >> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver > >> > [INFO] Unable to create class OMPMMul: falling back to java version > >> > { > >> > 0 => {0:26.0,1:35.0,2:46.0,3:51.0} > >> > 1 => {0:50.0,1:69.0,2:92.0,3:105.0} > >> > 2 => {0:62.0,1:86.0,2:115.0,3:132.0} > >> > 3 => {0:74.0,1:103.0,2:138.0,3:159.0} > >> > } > >> > - C = A %*% inCoreB > >> > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver > >> > [WARN] Unable to create class GPUMMul: attempting OpenMP version > >> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver > >> > [INFO] Unable to create class OMPMMul: falling back to java version > >> > { > >> > 0 => {0:26.0,1:35.0,2:46.0,3:51.0} > >> > 1 => {0:50.0,1:69.0,2:92.0,3:105.0} > >> > 2 => {0:62.0,1:86.0,2:115.0,3:132.0} > >> > 3 => {0:74.0,1:103.0,2:138.0,3:159.0} > >> > } > >> > > >> > Output of `clinfo`: > >> > $ clinfo > >> > Number of platforms 1 > >> > Platform Name NVIDIA CUDA > >> > Platform Vendor NVIDIA Corporation > >> > Platform Version OpenCL 1.2 CUDA > 8.0.0 > >> > Platform Profile FULL_PROFILE > >> > Platform Extensions > >> > cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics > >> > cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics > >> > cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing > >> > cl_nv_compiler_options cl_nv_device_attribute_query > cl_nv_pragma_unroll > >> > cl_nv_copy_opts > >> > Platform Extensions function suffix NV > >> > > >> > Platform Name NVIDIA CUDA > >> > Number of devices 1 > >> > Device Name GeForce GTX 750 Ti > >> > Device Vendor NVIDIA Corporation > >> > Device Vendor ID 0x10de > >> > Device Version OpenCL 1.2 CUDA > >> > Driver Version 367.44 > >> > Device OpenCL C Version OpenCL C 1.2 > >> > Device Type GPU > >> > Device Profile FULL_PROFILE > >> > Device Topology (NV) PCI-E, 01:00.0 > >> > Max compute units 5 > >> > Max clock frequency 1150MHz > >> > Compute Capability (NV) 5.0 > >> > Device Partition (core) > >> > Max number of sub-devices 1 > >> > Supported partition types None > >> > Max work item dimensions 3 > >> > Max work item sizes 1024x1024x64 > >> > Max work group size 1024 > >> > Preferred work group size multiple 32 > >> > Warp size (NV) 32 > >> > Preferred / native vector sizes > >> > char 1 / 1 > >> > short 1 / 1 > >> > int 1 / 1 > >> > long 1 / 1 > >> > half 0 / 0 > >> (n/a) > >> > float 1 / 1 > >> > double 1 / 1 > >> > (cl_khr_fp64) > >> > Half-precision Floating-point support (n/a) > >> > Single-precision Floating-point support (core) > >> > Denormals Yes > >> > Infinity and NANs Yes > >> > Round to nearest Yes > >> > Round to zero Yes > >> > Round to infinity Yes > >> > IEEE754-2008 fused multiply-add Yes > >> > Support is emulated in software No > >> > Correctly-rounded divide and sqrt operations Yes > >> > Double-precision Floating-point support (cl_khr_fp64) > >> > Denormals Yes > >> > Infinity and NANs Yes > >> > Round to nearest Yes > >> > Round to zero Yes > >> > Round to infinity Yes > >> > IEEE754-2008 fused multiply-add Yes > >> > Support is emulated in software No > >> > Correctly-rounded divide and sqrt operations No > >> > Address bits 64, Little-Endian > >> > Global memory size 2095841280 > (1.952GiB) > >> > Error Correction support No > >> > Max memory allocation 523960320 (499.7MiB) > >> > Unified memory for Host and Device No > >> > Integrated memory (NV) No > >> > Minimum alignment for any data type 128 bytes > >> > Alignment of base address 4096 bits (512 > bytes) > >> > Global Memory cache type Read/Write > >> > Global Memory cache size 81920 > >> > Global Memory cache line 128 bytes > >> > Image support Yes > >> > Max number of samplers per kernel 32 > >> > Max size for 1D images from buffer 134217728 pixels > >> > Max 1D or 2D image array size 2048 images > >> > Max 2D image size 16384x16384 pixels > >> > Max 3D image size 4096x4096x4096 > pixels > >> > Max number of read image args 256 > >> > Max number of write image args 16 > >> > Local memory type Local > >> > Local memory size 49152 (48KiB) > >> > Registers per block (NV) 65536 > >> > Max constant buffer size 65536 (64KiB) > >> > Max number of constant args 9 > >> > Max size of kernel argument 4352 (4.25KiB) > >> > Queue properties > >> > Out-of-order execution Yes > >> > Profiling Yes > >> > Prefer user sync for interop No > >> > Profiling timer resolution 1000ns > >> > Execution capabilities > >> > Run OpenCL kernels Yes > >> > Run native kernels No > >> > Kernel execution timeout (NV) Yes > >> > Concurrent copy and kernel execution (NV) Yes > >> > Number of async copy engines 1 > >> > printf() buffer size 1048576 (1024KiB) > >> > Built-in kernels > >> > Device Available Yes > >> > Compiler Available Yes > >> > Linker Available Yes > >> > Device Extensions > >> > cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics > >> > cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics > >> > cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing > >> > cl_nv_compiler_options cl_nv_device_attribute_query > cl_nv_pragma_unroll > >> > cl_nv_copy_opts > >> > > >> > NULL platform behavior > >> > clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) NVIDIA CUDA > >> > clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) Success [NV] > >> > clCreateContext(NULL, ...) [default] Success [NV] > >> > clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found > in > >> > platform > >> > clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) No platform > >> > clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No > devices > >> > found in platform > >> > clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) No devices > >> found in > >> > platform > >> > clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) No platform > >> > > >> > ICD loader properties > >> > ICD loader Name OpenCL ICD Loader > >> > ICD loader Vendor OCL Icd free > software > >> > ICD loader Version 2.2.8 > >> > ICD loader Profile OpenCL 1.2 > >> > NOTE: your OpenCL library declares to support OpenCL 1.2, > >> > but it seems to support up to OpenCL 2.1 too. > >> > > >> > > > > >
