That is weird.. did u tried it with Trevor's fix?


Sent from my Verizon Wireless 4G LTE smartphone


-------- Original message --------
From: Andrew Musselman <andrew.mussel...@gmail.com>
Date: 10/17/2016 10:11 AM (GMT-08:00)
To: dev@mahout.apache.org
Subject: Re: Trying out PR 261

I got the error before setting Xmx however.

On Mon, Oct 17, 2016 at 10:03 AM, Andrew Palumbo <ap....@outlook.com> wrote:

> I think that the memory should be set to 3 or  4G in the tests in
> pom.xml.. maybe you're o erring that setting?
>
>
>
> Sent from my Verizon Wireless 4G LTE smartphone
>
>
> -------- Original message --------
> From: Andrew Musselman <andrew.mussel...@gmail.com>
> Date: 10/17/2016 9:47 AM (GMT-08:00)
> To: dev@mahout.apache.org
> Subject: Re: Trying out PR 261
>
> Using the viennacl-omp profile I'm getting an out of memory error even with
> Xmx set in MAHOUT_OPTS to 1g:
>
> - mmul microbenchmark
>   + Mahout multiplication time: 8920 ms.
>   + ViennaCL/cpu/OpenMP multiplication time: 616 ms.
> - trans
> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> [WARN] Unable to create class GPUMMul: attempting OpenMP version
> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> [INFO] Unable to create class OMPMMul: falling back to java version
> *** RUN ABORTED ***
>   java.lang.OutOfMemoryError: Java heap space
>   at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.rehash(
> Int2DoubleOpenHashMap.java:1059)
>   at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.insert(
> Int2DoubleOpenHashMap.java:295)
>   at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.put(
> Int2DoubleOpenHashMap.java:301)
>   at
> org.apache.mahout.math.RandomAccessSparseVector.setQuick(
> RandomAccessSparseVector.java:130)
>   at
> org.apache.mahout.math.SparseRowMatrix.setQuick(SparseRowMatrix.java:105)
>   at org.apache.mahout.math.AbstractMatrix.assign(AbstractMatrix.java:256)
>   at
> org.apache.mahout.math.scalabindings.MatrixOps.$
> colon$eq(MatrixOps.scala:192)
>   at
> org.apache.mahout.math.scalabindings.MatrixOps.cloned(MatrixOps.scala:260)
>   at
> org.apache.mahout.math.scalabindings.MatrixOps.$minus(MatrixOps.scala:66)
>   at
> org.apache.mahout.viennacl.openmp.ViennaCLSuiteOMP$$
> anonfun$4.apply$mcV$sp(ViennaCLSuiteOMP.scala:152)
>   ...
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Mahout Build Tools ................................. SUCCESS [
>  1.131 s]
> [INFO] Apache Mahout ...................................... SUCCESS [
>  0.023 s]
> [INFO] Mahout Math ........................................ SUCCESS [
> 31.439 s]
> [INFO] Mahout HDFS ........................................ SUCCESS [
>  2.151 s]
> [INFO] Mahout Math Scala bindings ......................... SUCCESS [02:30
> min]
> [INFO] Mahout Spark bindings .............................. SUCCESS [
> 43.884 s]
> [INFO] Mahout Flink bindings .............................. SUCCESS [04:13
> min]
> [INFO] Mahout Spark bindings shell ........................ SUCCESS [
>  4.071 s]
> [INFO] Mahout Release Package ............................. SUCCESS [
>  1.311 s]
> [INFO] Mahout H2O backend ................................. SUCCESS [02:50
> min]
> [INFO] Mahout Native VienniaCL OpenMP Bindings ............ FAILURE [
> 36.106 s]
>
> On Sun, Oct 16, 2016 at 9:43 PM, Trevor Grant <trevor.d.gr...@gmail.com>
> wrote:
>
> > NOTE:
> >
> > The last solution creates a cyclical dependency-
> >
> > The way I made it work was happen-stance.
> >
> > 1)
> > mvn clean install -Pviennacl -Phadoop2 -DskipTests
> >
> > 2)
> > update the pom as described previously
> >
> > 3)
> > cd math-scala
> >
> > 4)
> > mvn test
> >
> > that will make it work- hacky but POC.
> >
> > Next how to do this 'right'.
> >
> > tg
> >
> >
> >
> > Trevor Grant
> > Data Scientist
> > https://github.com/rawkintrevo
> > http://stackexchange.com/users/3002022/rawkintrevo
> > http://trevorgrant.org
> >
> > *"Fortunate is he, who is able to know the causes of things."  -Virgil*
> >
> >
> > On Sun, Oct 16, 2016 at 11:29 PM, Trevor Grant <trevor.d.gr...@gmail.com
> >
> > wrote:
> >
> > > Maybe not the long term fix, but adding
> > >
> > > <dependency>
> > >      <groupId>org.apache.mahout</groupId>
> > >      <artifactId>mahout-native-viennacl_${scala.compat.
> > > version}</artifactId>
> > >      <scope>test</scope>
> > >    </dependency>
> > >
> > > to math-scala/pom.xml seems to fix the afore mentioned errors
> > >
> > >
> > > Trevor Grant
> > > Data Scientist
> > > https://github.com/rawkintrevo
> > > http://stackexchange.com/users/3002022/rawkintrevo
> > > http://trevorgrant.org
> > >
> > > *"Fortunate is he, who is able to know the causes of things."  -Virgil*
> > >
> > >
> > > On Sun, Oct 16, 2016 at 10:23 PM, Andrew Palumbo <ap....@outlook.com>
> > > wrote:
> > >
> > >> Thanks Trevor and Andrew- I think that both of your environments are
> > >> fine.. (Ubuntu 16.04 was just to recommended because ViennaCL 1.7.1 is
> > >> easily installed with sudo-apt-get install.  As logn as you have 1.7.1
> > on
> > >> the correct path, you should be fine.
> > >>
> > >>
> > >> I'm getting these errors now also, and by viewing the stack trace, I
> can
> > >> see that the mahout_native_vienniacl*.jars did not make it onto the
> > >> classpath some how:
> > >>
> > >>
> > >> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> > >> [WARN] Unable to create class GPUMMul: attempting OpenMP version
> > >> java.lang.ClassNotFoundException: org.apache.mahout.viennacl.ope
> > >> ncl.GPUMMul$
> > >>     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > >>     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > >>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > >>     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > >>     at java.lang.Class.forName0(Native Method)
> > >>     at java.lang.Class.forName(Class.java:264)
> > >>     at org.apache.mahout.math.scalabindings.SolverFactory$.getOpera
> > >> tor(SolverFactory.scala:50)
> > >>
> > >>
> > >> I think that i might have merged some conflicts badly when I rebased
> > last
> > >> week, because I was not seeing these errors.  I'll tey to get this
> > worked
> > >> out in the next couple of days.
> > >>
> > >>
> > >>
> > >> ________________________________
> > >> From: Trevor Grant <trevor.d.gr...@gmail.com>
> > >> Sent: Sunday, October 16, 2016 11:04:46 PM
> > >> To: dev@mahout.apache.org
> > >> Subject: Re: Trying out PR 261
> > >>
> > >> Having similar issues-
> > >>
> > >> Running 14.04 (I know, supposed to be 16.04, but whatever bear with
> me)
> > >>
> > >> Downloaded viennacl source, unzipped tar-dir/viennacl to
> > >> /usr/include/viennacl (e.g. 'installed' viennacl 1.7.1)
> > >>
> > >> Updated gcc (though I don't think this was neccessary, now that I see
> > >> Andrew M was having the same issues)
> > >> sudo add-apt-repository ppa:ubuntu-toolchain-r/test
> > >> sudo apt-get update
> > >> sudo apt-get install gcc-5 g++-5
> > >>
> > >> sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 0
> > >> --slave /usr/bin/g++ g++ /usr/bin/g++-5
> > >>
> > >>
> > >> System / GPU info
> > >> +------------------------------------------------------+
> > >>
> > >> | NVIDIA-SMI 352.63     Driver Version: 352.63         |
> > >>
> > >> |-------------------------------+----------------------+----
> > >> ------------------+
> > >> | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile
> > Uncorr.
> > >> ECC |
> > >> | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util
> > Compute
> > >> M. |
> > >> |===============================+======================+====
> > >> ==================|
> > >> |   0  GeForce GT 740      Off  | 0000:02:00.0     N/A |
> > >>  N/A |
> > >> | 33%   34C    P8    N/A /  N/A |    411MiB /  1021MiB |     N/A
> > >>  Default |
> > >> +-------------------------------+----------------------+----
> > >> ------------------+
> > >>
> > >>
> > >>
> > >>
> > >> Build was successful- no problems there.
> > >>
> > >> Still getting a slew of:
> > >>
> > >> [32mMatlabLikeMatrixOpsSuite: [0m
> > >> [32m- multiplication [0m
> > >> [32m- Hadamard [0m
> > >> [32mRLikeVectorOpsSuite: [0m
> > >> [32m- Hadamard [0m
> > >> 19489 [ScalaTest-main-running-RLikeVectorOpsSuite] DEBUG
> > >> org.apache.mahout.math.scalabindings.RLikeVectorOpsSuite  - dense
> > vector
> > >> dots:127.00 ms.
> > >> 19489 [ScalaTest-main-running-RLikeVectorOpsSuite] DEBUG
> > >> org.apache.mahout.math.scalabindings.RLikeVectorOpsSuite  - dense
> view
> > >> dots:201.00 ms.
> > >> [32m- dot-view performance [0m
> > >> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> > >> [WARN] Unable to create class GPUMMul: attempting OpenMP version
> > >> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> > >> [INFO] Unable to create class OMPMMul: falling back to java version
> > >> 19491 [ScalaTest-main-running-MathSuite] DEBUG
> > >> org.apache.mahout.math.scalabindings.MMul$  - AAt computation
> detected.
> > >> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> > >> [WARN] Unable to create class GPUMMul: attempting OpenMP version
> > >> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> > >> [INFO] Unable to create class OMPMMul: falling back to java version
> > >> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> > >> [WARN] Unable to create class GPUMMul: attempting OpenMP version
> > >> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> > >> [INFO] Unable to create class OMPMMul: falling back to java version
> > >> ... goes on like that for a long long time
> > >>
> > >>
> > >>
> > >> Trevor Grant
> > >> Data Scientist
> > >> https://github.com/rawkintrevo
> > >> http://stackexchange.com/users/3002022/rawkintrevo
> > >> http://trevorgrant.org
> > >>
> > >> *"Fortunate is he, who is able to know the causes of things."
> -Virgil*
> > >>
> > >>
> > >> On Sun, Oct 16, 2016 at 6:59 PM, Andrew Musselman <
> > >> andrew.mussel...@gmail.com> wrote:
> > >>
> > >> > I'm not sure my setup is correct; on Ubuntu 16.04 I'm getting a
> > >> firehose of
> > >> > infos and warns about not finding the GPUMMul and OMPMMul classes.
> > I've
> > >> > installed libviennacl-dev 1.7.1 through apt-get. Any ideas?
> > >> >
> > >> > Here's repro:
> > >> > $ git fetch origin pull/261/head:viennacl-opmmul-a
> > >> > $ mvn clean install -Pviennacl -Phadoop2
> > >> >
> > >> > Infos and warns (these are littered consistently throughout test
> > logs):
> > >> > %*% B mapBlock {}
> > >> > - C = A %*% B incompatible B keys
> > >> > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> > >> > [WARN] Unable to create class GPUMMul: attempting OpenMP version
> > >> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> > >> > [INFO] Unable to create class OMPMMul: falling back to java version
> > >> > - Spark-specific C = At %*% B , join
> > >> > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> > >> > [WARN] Unable to create class GPUMMul: attempting OpenMP version
> > >> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> > >> > [INFO] Unable to create class OMPMMul: falling back to java version
> > >> > - C = At %*% B , join, String-keyed
> > >> > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> > >> > [WARN] Unable to create class GPUMMul: attempting OpenMP version
> > >> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> > >> > [INFO] Unable to create class OMPMMul: falling back to java version
> > >> > - C = At %*% B , zippable, String-keyed
> > >> > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> > >> > [WARN] Unable to create class GPUMMul: attempting OpenMP version
> > >> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> > >> > [INFO] Unable to create class OMPMMul: falling back to java version
> > >> > - C = A %*% B.t
> > >> > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> > >> > [WARN] Unable to create class GPUMMul: attempting OpenMP version
> > >> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> > >> > [INFO] Unable to create class OMPMMul: falling back to java version
> > >> > {
> > >> >  0 => {0:26.0,1:35.0,2:46.0,3:51.0}
> > >> >  1 => {0:50.0,1:69.0,2:92.0,3:105.0}
> > >> >  2 => {0:62.0,1:86.0,2:115.0,3:132.0}
> > >> >  3 => {0:74.0,1:103.0,2:138.0,3:159.0}
> > >> > }
> > >> > - C = A %*% inCoreB
> > >> > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> > >> > [WARN] Unable to create class GPUMMul: attempting OpenMP version
> > >> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> > >> > [INFO] Unable to create class OMPMMul: falling back to java version
> > >> > {
> > >> >  0 => {0:26.0,1:35.0,2:46.0,3:51.0}
> > >> >  1 => {0:50.0,1:69.0,2:92.0,3:105.0}
> > >> >  2 => {0:62.0,1:86.0,2:115.0,3:132.0}
> > >> >  3 => {0:74.0,1:103.0,2:138.0,3:159.0}
> > >> > }
> > >> >
> > >> > Output of `clinfo`:
> > >> > $ clinfo
> > >> > Number of platforms                               1
> > >> >   Platform Name                                   NVIDIA CUDA
> > >> >   Platform Vendor                                 NVIDIA Corporation
> > >> >   Platform Version                                OpenCL 1.2 CUDA
> > 8.0.0
> > >> >   Platform Profile                                FULL_PROFILE
> > >> >   Platform Extensions
> > >> > cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_
> atomics
> > >> > cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics
> > >> > cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd
> cl_khr_gl_sharing
> > >> > cl_nv_compiler_options cl_nv_device_attribute_query
> > cl_nv_pragma_unroll
> > >> > cl_nv_copy_opts
> > >> >   Platform Extensions function suffix             NV
> > >> >
> > >> >   Platform Name                                   NVIDIA CUDA
> > >> > Number of devices                                 1
> > >> >   Device Name                                     GeForce GTX 750 Ti
> > >> >   Device Vendor                                   NVIDIA Corporation
> > >> >   Device Vendor ID                                0x10de
> > >> >   Device Version                                  OpenCL 1.2 CUDA
> > >> >   Driver Version                                  367.44
> > >> >   Device OpenCL C Version                         OpenCL C 1.2
> > >> >   Device Type                                     GPU
> > >> >   Device Profile                                  FULL_PROFILE
> > >> >   Device Topology (NV)                            PCI-E, 01:00.0
> > >> >   Max compute units                               5
> > >> >   Max clock frequency                             1150MHz
> > >> >   Compute Capability (NV)                         5.0
> > >> >   Device Partition                                (core)
> > >> >     Max number of sub-devices                     1
> > >> >     Supported partition types                     None
> > >> >   Max work item dimensions                        3
> > >> >   Max work item sizes                             1024x1024x64
> > >> >   Max work group size                             1024
> > >> >   Preferred work group size multiple              32
> > >> >   Warp size (NV)                                  32
> > >> >   Preferred / native vector sizes
> > >> >     char                                                 1 / 1
> > >> >     short                                                1 / 1
> > >> >     int                                                  1 / 1
> > >> >     long                                                 1 / 1
> > >> >     half                                                 0 / 0
> > >> (n/a)
> > >> >     float                                                1 / 1
> > >> >     double                                               1 / 1
> > >> >  (cl_khr_fp64)
> > >> >   Half-precision Floating-point support           (n/a)
> > >> >   Single-precision Floating-point support         (core)
> > >> >     Denormals                                     Yes
> > >> >     Infinity and NANs                             Yes
> > >> >     Round to nearest                              Yes
> > >> >     Round to zero                                 Yes
> > >> >     Round to infinity                             Yes
> > >> >     IEEE754-2008 fused multiply-add               Yes
> > >> >     Support is emulated in software               No
> > >> >     Correctly-rounded divide and sqrt operations  Yes
> > >> >   Double-precision Floating-point support         (cl_khr_fp64)
> > >> >     Denormals                                     Yes
> > >> >     Infinity and NANs                             Yes
> > >> >     Round to nearest                              Yes
> > >> >     Round to zero                                 Yes
> > >> >     Round to infinity                             Yes
> > >> >     IEEE754-2008 fused multiply-add               Yes
> > >> >     Support is emulated in software               No
> > >> >     Correctly-rounded divide and sqrt operations  No
> > >> >   Address bits                                    64, Little-Endian
> > >> >   Global memory size                              2095841280
> > (1.952GiB)
> > >> >   Error Correction support                        No
> > >> >   Max memory allocation                           523960320
> (499.7MiB)
> > >> >   Unified memory for Host and Device              No
> > >> >   Integrated memory (NV)                          No
> > >> >   Minimum alignment for any data type             128 bytes
> > >> >   Alignment of base address                       4096 bits (512
> > bytes)
> > >> >   Global Memory cache type                        Read/Write
> > >> >   Global Memory cache size                        81920
> > >> >   Global Memory cache line                        128 bytes
> > >> >   Image support                                   Yes
> > >> >     Max number of samplers per kernel             32
> > >> >     Max size for 1D images from buffer            134217728 pixels
> > >> >     Max 1D or 2D image array size                 2048 images
> > >> >     Max 2D image size                             16384x16384 pixels
> > >> >     Max 3D image size                             4096x4096x4096
> > pixels
> > >> >     Max number of read image args                 256
> > >> >     Max number of write image args                16
> > >> >   Local memory type                               Local
> > >> >   Local memory size                               49152 (48KiB)
> > >> >   Registers per block (NV)                        65536
> > >> >   Max constant buffer size                        65536 (64KiB)
> > >> >   Max number of constant args                     9
> > >> >   Max size of kernel argument                     4352 (4.25KiB)
> > >> >   Queue properties
> > >> >     Out-of-order execution                        Yes
> > >> >     Profiling                                     Yes
> > >> >   Prefer user sync for interop                    No
> > >> >   Profiling timer resolution                      1000ns
> > >> >   Execution capabilities
> > >> >     Run OpenCL kernels                            Yes
> > >> >     Run native kernels                            No
> > >> >     Kernel execution timeout (NV)                 Yes
> > >> >   Concurrent copy and kernel execution (NV)       Yes
> > >> >     Number of async copy engines                  1
> > >> >   printf() buffer size                            1048576 (1024KiB)
> > >> >   Built-in kernels
> > >> >   Device Available                                Yes
> > >> >   Compiler Available                              Yes
> > >> >   Linker Available                                Yes
> > >> >   Device Extensions
> > >> > cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_
> atomics
> > >> > cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics
> > >> > cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd
> cl_khr_gl_sharing
> > >> > cl_nv_compiler_options cl_nv_device_attribute_query
> > cl_nv_pragma_unroll
> > >> > cl_nv_copy_opts
> > >> >
> > >> > NULL platform behavior
> > >> >   clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  NVIDIA CUDA
> > >> >   clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   Success [NV]
> > >> >   clCreateContext(NULL, ...) [default]            Success [NV]
> > >> >   clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices
> found
> > in
> > >> > platform
> > >> >   clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  No platform
> > >> >   clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No
> > devices
> > >> > found in platform
> > >> >   clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  No devices
> > >> found in
> > >> > platform
> > >> >   clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  No platform
> > >> >
> > >> > ICD loader properties
> > >> >   ICD loader Name                                 OpenCL ICD Loader
> > >> >   ICD loader Vendor                               OCL Icd free
> > software
> > >> >   ICD loader Version                              2.2.8
> > >> >   ICD loader Profile                              OpenCL 1.2
> > >> > NOTE: your OpenCL library declares to support OpenCL 1.2,
> > >> > but it seems to support up to OpenCL 2.1 too.
> > >> >
> > >>
> > >
> > >
> >
>

Reply via email to