subject:"\[jira\] \[Commented\] \(MAHOUT\-1885\) Inital Implementation of VCL Bindings"

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2017-01-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839198#comment-15839198
 ] 

Hudson commented on MAHOUT-1885:


FAILURE: Integrated in Jenkins build Mahout-Quality #3410 (See 
[https://builds.apache.org/job/Mahout-Quality/3410/])
MAHOUT-1885: Inital commit of VCL bindings. closes apache/mahout#269 (apalumbo: 
rev 034790cce40fcee2b7a875c482345d35f7c0fa8d)
* (add) viennacl/pom.xml
* (edit) distribution/pom.xml
* (edit) 
spark/src/test/scala/org/apache/mahout/drivers/RowSimilarityDriverSuite.scala
* (add) 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/VCLVector.scala
* (edit) spark/src/main/scala/org/apache/mahout/sparkbindings/package.scala
* (add) 
viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/ProdExpression.scala
* (edit) 
spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala
* (add) 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/VectorBase.scala
* (edit) flink/pom.xml
* (add) 
viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/VecMultExpression.scala
* (add) 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/MatVecProdExpression.scala
* (add) 
viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/SrMatDnMatProdExpression.scala
* (add) 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/package.scala
* (edit) runtests.sh
* (add) 
viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/Context.scala
* (add) 
math-scala/src/main/scala/org/apache/mahout/math/backend/jvm/JvmBackend.scala
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/GPUMMul.scala
* (add) viennacl-omp/runs
* (add) 
viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/VectorBase.scala
* (add) 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/CompressedMatrix.scala
* (add) 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/ProdExpression.scala
* (add) 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/SrMatDnMatProdExpression.scala
* (add) 
viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/VCLVector.scala
* (edit) 
spark/src/test/scala/org/apache/mahout/sparkbindings/test/DistributedSparkSuite.scala
* (edit) 
spark/src/test/scala/org/apache/mahout/sparkbindings/SparkBindingsSuite.scala
* (add) 
viennacl-omp/src/main/java/org/apache/mahout/viennacl/openmp/javacpp/LinalgFunctions.java
* (add) 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/MatMatProdExpression.scala
* (add) 
viennacl-omp/src/main/java/org/apache/mahout/viennacl/openmp/javacpp/MatrixTransExpression.scala
* (add) 
viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/MatMatProdExpression.scala
* (edit) pom.xml
* (add) 
viennacl/src/main/java/org/apache/mahout/viennacl/opencl/javacpp/MatrixTransExpression.scala
* (add) 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/OMPMMul.scala
* (add) 
viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/MemHandle.scala
* (add) math-scala/src/main/scala/org/apache/mahout/math/backend/Backend.scala
* (edit) 
math-scala/src/test/scala/org/apache/mahout/math/scalabindings/RLikeMatrixOpsSuite.scala
* (add) 
math-scala/src/main/scala/org/apache/mahout/math/backend/RootSolverFactory.scala
* (edit) bin/mahout
* (edit) spark/pom.xml
* (add) 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/VecMultExpression.scala
* (add) 
math-scala/src/main/scala/org/apache/mahout/math/backend/incore/package.scala
* (edit) spark-shell/pom.xml
* (add) 
viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/DenseColumnMatrix.scala
* (add) viennacl-omp/linux-haswell.properties
* (add) viennacl/linux-x86_64-viennacl.properties
* (add) 
viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/javacpp/MatVecProdExpression.scala
* (edit) spark/src/main/assembly/dependency-reduced.xml
* (add) 
viennacl-omp/src/main/java/org/apache/mahout/viennacl/openmp/javacpp/Functions.java
* (add) viennacl/linux-haswell.properties
* (add) 
math-scala/src/test/scala/org/apache/mahout/math/backend/BackendSuite.scala
* (add) 
viennacl/src/test/scala/org/apache/mahout/viennacl/opencl/ViennaCLSuiteVCL.scala
* (edit) 
spark/src/test/scala/org/apache/mahout/drivers/TextDelimitedReaderWriterSuite.scala
* (add) 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/Context.scala
* (add) 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/MatrixBase.scala
* (add) 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/DenseRowMatrix.scala
* (edit) .travis.yml
* (add) viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/package.scala
* (add) 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/javacpp/MemHandle.scala
* (add)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2017-01-25 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839158#comment-15839158
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user asfgit closed the pull request at:

https://github.com/apache/mahout/pull/261


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2017-01-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824871#comment-15824871
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r96336410
  
--- Diff: LICENSE.txt ---
@@ -799,3 +799,32 @@ The following license applies to the H2O package
   identification within third-party archives.
 
Copyright 2012 0xdata, Inc
+
+
+The following applies to the ViennaCL library and files in the 
mahout-native-viennacl module
+
+
+   Copyright (c) 2010-2016 Institute for Microelectronics,
+   Institute for Analysis and Scientific 
Computing, TU Wien.
+   Portions of this software are copyright by UChicago Argonne, LLC.
+   Argonne National Laboratory, with facilities in the state of Illinois,
+   is owned by The United States Government, and operated by UChicago 
Argonne, LLC
+   under provision of a contract with the Department of Energy.
+
+   Permission is hereby granted, free of charge, to any person obtaining a 
copy
+   of this software and associated documentation files (the "Software"), 
to deal
+   in the Software without restriction, including without limitation the 
rights
+   to use, copy, modify, merge, publish, distribute, sublicense, and/or 
sell
+   copies of the Software, and to permit persons to whom the Software is
+   furnished to do so, subject to the following conditions:
+
+   The above copyright notice and this permission notice shall be included 
in
+   all copies or substantial portions of the Software.
+
+   THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS 
OR
+   IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+   FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL 
THE
+   AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+   LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 
FROM,
+   OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
IN
+   THE SOFTWARE.
--- End diff --

Done, thanks @smarthi - not sure why this is not showing up as out of date.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2017-01-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824869#comment-15824869
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r96336350
  
--- Diff: 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/OMPMMul.scala ---
@@ -0,0 +1,448 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mahout.viennacl.openmp
+
+import org.apache.mahout.logging._
+import org.apache.mahout.math
+import org.apache.mahout.math._
+import org.apache.mahout.math.flavor.{BackEnum, TraversingStructureEnum}
+import org.apache.mahout.math.function.Functions
+import org.apache.mahout.math.scalabindings.RLikeOps._
+import org.apache.mahout.math.scalabindings._
+import org.apache.mahout.viennacl.openmp.javacpp.Functions._
+import org.apache.mahout.viennacl.openmp.javacpp.LinalgFunctions._
+import org.apache.mahout.viennacl.openmp.javacpp.{CompressedMatrix, 
Context, DenseRowMatrix}
+
+import scala.collection.JavaConversions._
+
+object OMPMMul extends MMBinaryFunc {
+
+  private final implicit val log = getLog(OMPMMul.getClass)
+
+  override def apply(a: Matrix, b: Matrix, r: Option[Matrix]): Matrix = {
+
+require(a.ncol == b.nrow, "Incompatible matrix sizes in matrix 
multiplication.")
+
+val (af, bf) = (a.getFlavor, b.getFlavor)
+val backs = (af.getBacking, bf.getBacking)
+val sd = (af.getStructure, math.scalabindings.densityAnalysis(a), 
bf.getStructure, densityAnalysis(b))
+
+
+try {
+
+  val alg: MMulAlg = backs match {
+
+// Both operands are jvm memory backs.
+case (BackEnum.JVMMEM, BackEnum.JVMMEM) ⇒
+
+  sd match {
+
+// Multiplication cases by a diagonal matrix.
+case (TraversingStructureEnum.VECTORBACKED, _, 
TraversingStructureEnum.COLWISE, _)
+  if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagCW
+case (TraversingStructureEnum.VECTORBACKED, _, 
TraversingStructureEnum.SPARSECOLWISE, _)
+  if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagCW
+case (TraversingStructureEnum.VECTORBACKED, _, 
TraversingStructureEnum.ROWWISE, _)
+  if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagRW
+case (TraversingStructureEnum.VECTORBACKED, _, 
TraversingStructureEnum.SPARSEROWWISE, _)
+  if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagRW
+
+case (TraversingStructureEnum.COLWISE, _, 
TraversingStructureEnum.VECTORBACKED, _)
+  if b.isInstanceOf[DiagonalMatrix] ⇒ jvmCWDiag
+case (TraversingStructureEnum.SPARSECOLWISE, _, 
TraversingStructureEnum.VECTORBACKED, _)
+  if b.isInstanceOf[DiagonalMatrix] ⇒ jvmCWDiag
+case (TraversingStructureEnum.ROWWISE, _, 
TraversingStructureEnum.VECTORBACKED, _)
+  if b.isInstanceOf[DiagonalMatrix] ⇒ jvmRWDiag
+case (TraversingStructureEnum.SPARSEROWWISE, _, 
TraversingStructureEnum.VECTORBACKED, _)
+  if b.isInstanceOf[DiagonalMatrix] ⇒ jvmRWDiag
+
+// Dense-dense cases
+case (TraversingStructureEnum.ROWWISE, true, 
TraversingStructureEnum.COLWISE, true) if a eq b.t ⇒ ompDRWAAt
+case (TraversingStructureEnum.ROWWISE, true, 
TraversingStructureEnum.COLWISE, true) if a.t eq b ⇒ ompDRWAAt
+case (TraversingStructureEnum.ROWWISE, true, 
TraversingStructureEnum.COLWISE, true) ⇒ ompRWCW
+case (TraversingStructureEnum.ROWWISE, true, 
TraversingStructureEnum.ROWWISE, true) ⇒ jvmRWRW
+case (TraversingStructureEnum.COLWISE, true, 
TraversingStructureEnum.COLWISE, true) ⇒ jvmCWCW
+case (TraversingStructureEnum.COLWISE, true, 
TraversingStructureEnum.ROWWISE, true) if a eq

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2017-01-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824866#comment-15824866
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r96336137
  
--- Diff: 
spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala
 ---
@@ -162,7 +163,7 @@ class MahoutSparkILoop extends SparkILoop {
  _ __ ___   __ _| |__   ___  _   _| |_
 | '_ ` _ \ / _` | '_ \ / _ \| | | | __|
 | | | | | | (_| | | | | (_) | |_| | |_
-|_| |_| |_|\__,_|_| |_|\___/ \__,_|\__|  version 0.12.2
+|_| |_| |_|\__,_|_| |_|\___/ \__,_|\__|  version 0.13.0
 
--- End diff --

+1


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2017-01-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824865#comment-15824865
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r96336102
  
--- Diff: 
viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/GPUMMul.scala ---
@@ -0,0 +1,455 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mahout.viennacl.opencl
+
+import org.apache.mahout.logging._
+import org.apache.mahout.math
+import org.apache.mahout.math._
+import org.apache.mahout.math.flavor.{BackEnum, TraversingStructureEnum}
+import org.apache.mahout.math.function.Functions
+import org.apache.mahout.math.scalabindings.RLikeOps._
+import org.apache.mahout.math.scalabindings._
+import org.apache.mahout.viennacl.opencl.javacpp.Functions._
+import org.apache.mahout.viennacl.opencl.javacpp.LinalgFunctions._
+import org.apache.mahout.viennacl.opencl.javacpp.{CompressedMatrix, 
Context, DenseRowMatrix}
+
+import scala.collection.JavaConversions._
+
+object GPUMMul extends MMBinaryFunc {
+
+  private final implicit val log = getLog(GPUMMul.getClass)
+
+  override def apply(a: Matrix, b: Matrix, r: Option[Matrix]): Matrix = {
+
+require(a.ncol == b.nrow, "Incompatible matrix sizes in matrix 
multiplication.")
+
+val (af, bf) = (a.getFlavor, b.getFlavor)
+val backs = (af.getBacking, bf.getBacking)
+val sd = (af.getStructure, math.scalabindings.densityAnalysis(a), 
bf.getStructure, densityAnalysis(b))
+
+
+try {
+
+  val alg: MMulAlg = backs match {
+
+// Both operands are jvm memory backs.
+case (BackEnum.JVMMEM, BackEnum.JVMMEM) ⇒
--- End diff --

Agreed (obviously this is an old comment due to my underestimation of what 
a move across the country and a new job can do to you +  a nephew destroying my 
Mahout machine)- I don't like keeping both modules around but, quoting myself 
from 6 months ago:

> We may be able to get away with only one Viennacl module by setting an 
environment variable and checking for it in Context.scala if it exists we 
"#define VIENNACL_WITH_OPENCL" which will make the module OpenMP only. The 
Problem With This Is That We Can't Test GPU in this module.. without checking 
for the environment var.

I guess the above is a possibility, but not especially simple.  For no, 
though all tests are passing.. though i am seeing that some of the eg. spark 
tests are not running `GPUMmul` when Built for `-Pviennacl`.  So I may need to 
fix some imports somewhere.  

Either way I 'd like to get the base Architecture nailed down with this 
release.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2017-01-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824862#comment-15824862
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261
  
All tests now pass in GPU and OpenMP mode.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2017-01-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824821#comment-15824821
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261
  
Interesting.. After fixing a few small bugs in the `viennacl-omp` module 
(00fbe3e),  all tests  pass with the exception of  `sparse mmul 
microbenchmark`, which fails with a 
``` java.lang.OutOfMemoryError: Java heap space
  at it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.rehash
```

if I chage the dimensions of the matrices to :
```  
val m = 300
val n = 300
val s = 10
```
(one order of magnitude lower on `m` and `n` two lower on `s`, all tests 
will pass:
```
Discovery completed in 296 milliseconds.
Run starting. Expected test count is: 6
ViennaCLSuiteOMP:
- row-major viennacl::matrix
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[WARN] Unable to create class GPUMMul: attempting OpenMP version
[INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.openmp.OMPMMul solver
jvmRWRW
ompRWCW
log4j:WARN No appenders could be found for logger 
(org.apache.mahout.viennacl.openmp.OMPMMul$).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for 
more info.
- mmul microbenchmark
  + Mahout multiplication time: 11444 ms. 
  + ViennaCL/cpu/OpenMP multiplication time: 2820 ms. 
- trans
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[WARN] Unable to create class GPUMMul: attempting OpenMP version
[INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.openmp.OMPMMul solver
ompSparseRWRW
- sparse mmul microbenchmark
  + Mahout Sparse multiplication time: 65 ms. 
  + ViennaCL/cpu/OpenMP Sparse multiplication time: 5 ms. 
- VCL Dense Matrix %*% Dense vector - no OpenCl
  + Mahout dense matrix %*% dense vector multiplication time: 30 ms. 
  + ViennaCL/cpu/OpenMP dense matrix %*% dense vector multiplication time: 
16 ms. 
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[WARN] Unable to create class GPUMMul: attempting OpenMP version
[INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.openmp.OMPMMul solver
ompSparseRowRWRW
- Sparse %*% Dense mmul microbenchmark
  + Mahout multiplication time: 2583 ms. 
  + ViennaCL/cpu/OpenMP multiplication time: 2639 ms. 
Run completed in 21 seconds, 602 milliseconds.
Total number of tests run: 6
Suites: completed 2, aborted 0
Tests: succeeded 6, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
[INFO] 

[INFO] BUILD SUCCESS
[INFO] 

[INFO] Total time: 40.433 s
[INFO] Finished at: 2017-01-16T17:27:49-08:00
[INFO] Final Memory: 21M/327M
[INFO] 

```

What's confusing to me is why they would pass on the GPU module, and not on 
the OpenMP module, which use almost Identical methods to convert back and forth 
to interface with ViennaCL operations.  As well, this same test is run for 
comparison in the GPU module.  What an annoying little bug.  Must be a maven 
test setting.  


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2017-01-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824609#comment-15824609
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261
  
I can now verify that from a fresh install of Ubuntu 16.04(.1)LTS, with the 
 Ubuntu-Provided NVIDIA-367 Drivers that the GPU section of this PR works 
without Issue, following the above instructions:
```
 - sparse mmul microbenchmark
  + Mahout Sparse multiplication time: 12314 ms. 
  + ViennaCL/OpenCL Sparse multiplication time: 10959 ms. 
  + ViennaCL/cpu/OpenMP Sparse multiplication time: 3690 ms. 
- VCL Dense Matrix %*% Dense vector
  + Mahout dense matrix %*% dense vector multiplication time: 0 ms. 
  + ViennaCL/cpu/OpenMP dense matrix %*% dense vector multiplication time: 
4 ms. 
```


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-11-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15682482#comment-15682482
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261
  
ok fixed up the poms and added profiles for viennacl/viennacl-omp in build 
pom.xml.  There is still something wrong with the viennacl-omp module, I'm not 
sure what but it is not recognized by the other poms, and the jar is not 
gettting picked up when its being run for tests.

Should be an easy fix, and I'll come back to it soon.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582758#comment-15582758
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261
  
I don't think  that cyclical referencing is an issue.  There are other ways 
to get the class onto the path.  They just got dropped somehow when I was 
creating/reason the pr.  For now though adding the dependency at test time 
works for me just to make sure everything's working.



Sent from my Verizon Wireless 4G LTE smartphone


 Original message 
From: Trevor Grant 
Date: 10/16/2016 9:54 PM (GMT-08:00)
To: apache/mahout 
Cc: Andrew Palumbo , Author 
Subject: Re: [apache/mahout] MAHOUT-1885 [WIP][FOR COMMENT]: Inital 
implementation of VCL bindings for mahout math. (#261)


@rawkintrevo commented on this pull request.



In 
math-scala/src/main/scala/org/apache/mahout/math/scalabindings/SolverFactory.scala:

> +  // just temp for quick POC
+  val classMap: Map[String,String] =
+Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
+("OMPMMul"->"org.apache.mahout.viennacl.openmp.OMPMMul"))
+}
+object SolverFactory extends SolverFactory {
+
+// default is JVM
+var clazz: MMBinaryFunc = MMul
+
+// eventually match on implicit Classtag . for now.  just take as is.
+// this is a bit hacky, Shoud not be doing onlytry/catch here..
+def getOperator[C: ClassTag]: MMBinaryFunc = {
+
+  try {
+println("creating org.apache.mahout.viennacl.opencl.GPUMMul 
solver")


given the cyclical reference issues, maybe a solution here should come 
sooner rather than later (e.g. solving this, would also 'fix' the cyclical 
reference problem?)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub, or mute the 
thread.



> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582746#comment-15582746
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261
  
I'd really opened this PR so that we could discuss architecture and the 
correct entry points for the SolverFactory, hence the ridiculously verbose and 
confusing debugging output.  I'd like to get the arch right before we commit.  
At its core it works as is, but there are alot of changes that need to be made.

Eg.  We may be able to get away with only one Viennacl module by setting an 
environment variable and checking for it in Context.scala if it exists we 
"#define VIENNACL_WITH_OPENCL" which will make the module OpenMP only.  The 
Problem With This Is That We Can't Test Gpu in this module.. without checking 
for the environment var.

The other big thing is that I don't know if the placement if the 
SolverFactory as it is is correct.. we may want to examine this, as well as 
cache a XXXMMUL Of both types the first time this class is hit so that we are 
not dynamically loading a class each time %*% is hit.

These are a couple of things that I'd like to get figured out before 
committing.

We've already identified a few pain points in the testing that everybody 
did over the weekend.  So let's get it right before committing.


Sent from my Verizon Wireless 4G LTE smartphone


 Original message 
From: Dmitriy Lyubimov 
Date: 10/17/2016 8:51 AM (GMT-08:00)
To: apache/mahout 
Cc: Andrew Palumbo , Author 
Subject: Re: [apache/mahout] MAHOUT-1885 [WIP][FOR COMMENT]: Inital 
implementation of VCL bindings for mahout math. (#261)


IMO this needs more work and committing this is premature at this point due 
to architecture considerations. Not only it misses some things, but also IMO 
has things that should not be done the way they currently are.

-
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub, or 
mute the 
thread.



> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582747#comment-15582747
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261
  
+1



Sent from my Verizon Wireless 4G LTE smartphone


 Original message 
From: Trevor Grant 
Date: 10/16/2016 9:53 PM (GMT-08:00)
To: apache/mahout 
Cc: Andrew Palumbo , Author 
Subject: Re: [apache/mahout] MAHOUT-1885 [WIP][FOR COMMENT]: Inital 
implementation of VCL bindings for mahout math. (#261)


@rawkintrevo commented on this pull request.



In 
spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala:

> @@ -162,7 +163,7 @@ class MahoutSparkILoop extends SparkILoop {
  _ __ ___   __ _| |__   ___  _   _| |_
 | '_ ` _ \ / _` | '_ \ / _ \| | | | __|
 | | | | | | (_| | | | | (_) | |_| | |_
-|_| |_| |_|\__,_|_| |_|\___/ \__,_|\__|  version 0.12.2
+|_| |_| |_|\__,_|_| |_|\___/ \__,_|\__|  version 0.13.0



I agree to keep. The Zeppelin integration has a few of its own little 
issues, esp when it comes to serialization, and some other stuff. The 
mahout-shell will always be way more 'bullet proof'. I also like to sanity 
check myself, to figure out is an issues 1) Zeppelin's Fault, 2) Mahout's Fault 
3) My Fault, by running code in the mahout-spark shell (after a zeppelin 
failure).

-
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub, or mute the 
thread.



> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582587#comment-15582587
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user dlyubimov commented on the issue:

https://github.com/apache/mahout/pull/261
  
IMO this needs more work and committing this is premature at this point due 
to architecture considerations. Not only it misses some things, but also IMO 
has things that should not be done the way they currently are.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-17 Thread Suneel Marthi (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582075#comment-15582075
 ] 

Suneel Marthi commented on MAHOUT-1885:
---

This has been fixed as of today in ViennaCL. Please update ur ViennaCL 
binaries. 

> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581210#comment-15581210
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user rawkintrevo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r83577666
  
--- Diff: 
math-scala/src/main/scala/org/apache/mahout/math/scalabindings/SolverFactory.scala
 ---
@@ -0,0 +1,71 @@
+/**
+  * Licensed to the Apache Software Foundation (ASF) under one or more
+  * contributor license agreements.  See the NOTICE file distributed with
+  * this work for additional information regarding copyright ownership.
+  * The ASF licenses this file to You under the Apache License, Version 2.0
+  * (the "License"); you may not use this file except in compliance with
+  * the License.  You may obtain a copy of the License at
+  *
+  * http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+package org.apache.mahout.math.scalabindings
+
+import java.io.File
+
+import org.apache.mahout.logging._
+
+import scala.reflect.ClassTag
+import scala.reflect.runtime._
+import scala.reflect._
+
+
+class SolverFactory {
+
+  private final implicit val log = getLog(this.getClass)
+
+  // just temp for quick POC
+  val classMap: Map[String,String] =
+Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
+("OMPMMul"->"org.apache.mahout.viennacl.openmp.OMPMMul"))
+}
+object SolverFactory extends SolverFactory {
+
+// default is JVM
+var clazz: MMBinaryFunc = MMul
+
+// eventually match on implicit Classtag . for now.  just take as is.
+// this is a bit hacky, Shoud not be doing onlytry/catch here..
+def getOperator[C: ClassTag]: MMBinaryFunc = {
+
+  try {
+println("creating org.apache.mahout.viennacl.opencl.GPUMMul 
solver")
--- End diff --

given the cyclical reference issues, maybe a solution here should come 
sooner rather than later (e.g. solving this, would also 'fix' the cyclical 
reference problem?)


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15580863#comment-15580863
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261
  
Correcting myself again,  This branch was built for *ViennaCL 1.7.1*.  

There is an error in the ViennaCL Library that shows the wrong version:

https://github.com/viennacl/viennacl-dev/blob/master/viennacl/version.hpp 


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-16 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15580526#comment-15580526
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261
  
I apologize - in the header and in the docs, I had VienniaCL 1.7.1+  this 
branch has been built for 1.7.0.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-11 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567552#comment-15567552
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82933987
  
--- Diff: 
spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala
 ---
@@ -162,7 +163,7 @@ class MahoutSparkILoop extends SparkILoop {
  _ __ ___   __ _| |__   ___  _   _| |_
 | '_ ` _ \ / _` | '_ \ / _ \| | | | __|
 | | | | | | (_| | | | | (_) | |_| | |_
-|_| |_| |_|\__,_|_| |_|\___/ \__,_|\__|  version 0.12.2
+|_| |_| |_|\__,_|_| |_|\___/ \__,_|\__|  version 0.13.0
 
--- End diff --

if it is not an expensive maintenance, i would vote to keep it. I actually 
occasionally use it as a scratchpad or a quick demo env


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-11 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567550#comment-15567550
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82934625
  
--- Diff: 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/OMPMMul.scala ---
@@ -0,0 +1,448 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mahout.viennacl.openmp
+
+import org.apache.mahout.logging._
+import org.apache.mahout.math
+import org.apache.mahout.math._
+import org.apache.mahout.math.flavor.{BackEnum, TraversingStructureEnum}
+import org.apache.mahout.math.function.Functions
+import org.apache.mahout.math.scalabindings.RLikeOps._
+import org.apache.mahout.math.scalabindings._
+import org.apache.mahout.viennacl.openmp.javacpp.Functions._
+import org.apache.mahout.viennacl.openmp.javacpp.LinalgFunctions._
+import org.apache.mahout.viennacl.openmp.javacpp.{CompressedMatrix, 
Context, DenseRowMatrix}
+
+import scala.collection.JavaConversions._
+
+object OMPMMul extends MMBinaryFunc {
--- End diff --

ok so likely it is WIP again, but we wanted a cache here. So i think on one 
hand it needs a specific proxy matrix type; on the other, it doesn't need all 
these mappings based on argument orientation. 

On the latter point in particualr, since we are going to a native solver, 
we are rewriting (copyign) data anyway, so we already have ok'd the fact that 
rewrite is less expensive than mmul itself. 

Many solvers will accept whatever orientation of arguments (can't 
immediately remember if it is the case with VCL OMP), and even if they do not, 
we just simply can copy it to whatever orientation is needed.

so i think all this matching eventually would be irrelevant really.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-11 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567551#comment-15567551
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82932910
  
--- Diff: 
math-scala/src/main/scala/org/apache/mahout/math/scalabindings/RLikeMatrixOps.scala
 ---
@@ -17,14 +17,22 @@
 package org.apache.mahout.math.scalabindings
 
 import org.apache.mahout.math.function.Functions
-import org.apache.mahout.math.{Vector, Matrix}
+import org.apache.mahout.math.{Matrix, Vector}
 import scala.collection.JavaConversions._
 import RLikeOps._
+import org.apache.mahout.math.scalabindings._
+
 
 class RLikeMatrixOps(m: Matrix) extends MatrixOps(m) {
 
   /** Structure-optimized mmul */
-  def %*%(that: Matrix) = MMul(m, that, None)
+
+  implicit var solverOperator: opMMulSolver =_
--- End diff --

Somehow i don't like implicit vars inside the decorator pattern. Clean 
decorator pattern (which is what it really is) assumes there's no new intrinsic 
state. so the only value instrinsic is `m`.

Instead, it should query solvers from a table as instantiated and saved in 
the current solver factory.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-11 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567547#comment-15567547
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82933150
  
--- Diff: 
math-scala/src/main/scala/org/apache/mahout/math/scalabindings/SolverFactory.scala
 ---
@@ -0,0 +1,71 @@
+/**
+  * Licensed to the Apache Software Foundation (ASF) under one or more
+  * contributor license agreements.  See the NOTICE file distributed with
+  * this work for additional information regarding copyright ownership.
+  * The ASF licenses this file to You under the Apache License, Version 2.0
+  * (the "License"); you may not use this file except in compliance with
+  * the License.  You may obtain a copy of the License at
+  *
+  * http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+package org.apache.mahout.math.scalabindings
+
+import java.io.File
+
+import org.apache.mahout.logging._
+
+import scala.reflect.ClassTag
+import scala.reflect.runtime._
+import scala.reflect._
+
+
+class SolverFactory {
+
+  private final implicit val log = getLog(this.getClass)
+
+  // just temp for quick POC
+  val classMap: Map[String,String] =
+Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
--- End diff --

no this class may not be present, there's no direct dependency on a 
concrete backend. This code is correct in its intent, but we will need a 
Backend interface (one for VCL, one for e.g., MKL etc.) that can probe & report 
availability of concrete solvers.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-11 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567554#comment-15567554
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82934105
  
--- Diff: 
spark/src/main/scala/org/apache/mahout/sparkbindings/package.scala ---
@@ -214,11 +223,11 @@ package object sparkbindings {
   // During maven tests, the maven classpath also creeps in for 
some reason
   !n.matches(".*/.m2/.*")
 )
-/* verify jar passed to context
-log.info("\n\n\n")
-mcjars.foreach(j => log.info(j))
-log.info("\n\n\n")
-*/
+/* verify jar passed to context */
+//info("\n\n\n")
--- End diff --

this IMO is a useful info (perhaps should be downgraded to debug and be 
made a little bit more verbose explaining what it is it prints)


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-11 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567546#comment-15567546
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82932953
  
--- Diff: 
math-scala/src/main/scala/org/apache/mahout/math/scalabindings/RLikeMatrixOps.scala
 ---
@@ -17,14 +17,22 @@
 package org.apache.mahout.math.scalabindings
 
 import org.apache.mahout.math.function.Functions
-import org.apache.mahout.math.{Vector, Matrix}
+import org.apache.mahout.math.{Matrix, Vector}
 import scala.collection.JavaConversions._
 import RLikeOps._
+import org.apache.mahout.math.scalabindings._
+
 
 class RLikeMatrixOps(m: Matrix) extends MatrixOps(m) {
 
   /** Structure-optimized mmul */
-  def %*%(that: Matrix) = MMul(m, that, None)
+
+  implicit var solverOperator: opMMulSolver =_
--- End diff --

PS one can look at Breeze for one way of doing this.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-11 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567548#comment-15567548
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82934828
  
--- Diff: 
viennacl/src/main/scala/org/apache/mahout/viennacl/opencl/GPUMMul.scala ---
@@ -0,0 +1,455 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mahout.viennacl.opencl
+
+import org.apache.mahout.logging._
+import org.apache.mahout.math
+import org.apache.mahout.math._
+import org.apache.mahout.math.flavor.{BackEnum, TraversingStructureEnum}
+import org.apache.mahout.math.function.Functions
+import org.apache.mahout.math.scalabindings.RLikeOps._
+import org.apache.mahout.math.scalabindings._
+import org.apache.mahout.viennacl.opencl.javacpp.Functions._
+import org.apache.mahout.viennacl.opencl.javacpp.LinalgFunctions._
+import org.apache.mahout.viennacl.opencl.javacpp.{CompressedMatrix, 
Context, DenseRowMatrix}
+
+import scala.collection.JavaConversions._
+
+object GPUMMul extends MMBinaryFunc {
+
+  private final implicit val log = getLog(GPUMMul.getClass)
+
+  override def apply(a: Matrix, b: Matrix, r: Option[Matrix]): Matrix = {
+
+require(a.ncol == b.nrow, "Incompatible matrix sizes in matrix 
multiplication.")
+
+val (af, bf) = (a.getFlavor, b.getFlavor)
+val backs = (af.getBacking, bf.getBacking)
+val sd = (af.getStructure, math.scalabindings.densityAnalysis(a), 
bf.getStructure, densityAnalysis(b))
+
+
+try {
+
+  val alg: MMulAlg = backs match {
+
+// Both operands are jvm memory backs.
+case (BackEnum.JVMMEM, BackEnum.JVMMEM) ⇒
--- End diff --

Same comment here as for the OMP solver. 

Also: I am still not convinced we need two actual different classes to 
support OMP and GPU. I thought we could use one concrete class that shares most 
of execution path for both cases, and just switch it to use OMP only or GPU. 

We then potentially could report/instantiate two instances of essentially 
the same class as solvers for OMP and GPU.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-11 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567549#comment-15567549
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82933830
  
--- Diff: 
math-scala/src/main/scala/org/apache/mahout/math/scalabindings/SolverFactory.scala
 ---
@@ -0,0 +1,71 @@
+/**
+  * Licensed to the Apache Software Foundation (ASF) under one or more
+  * contributor license agreements.  See the NOTICE file distributed with
+  * this work for additional information regarding copyright ownership.
+  * The ASF licenses this file to You under the Apache License, Version 2.0
+  * (the "License"); you may not use this file except in compliance with
+  * the License.  You may obtain a copy of the License at
+  *
+  * http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+package org.apache.mahout.math.scalabindings
+
+import java.io.File
+
+import org.apache.mahout.logging._
+
+import scala.reflect.ClassTag
+import scala.reflect.runtime._
+import scala.reflect._
+
+
+class SolverFactory {
+
+  private final implicit val log = getLog(this.getClass)
+
+  // just temp for quick POC
+  val classMap: Map[String,String] =
+Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
+("OMPMMul"->"org.apache.mahout.viennacl.openmp.OMPMMul"))
+}
+object SolverFactory extends SolverFactory {
+
+// default is JVM
+var clazz: MMBinaryFunc = MMul
+
+// eventually match on implicit Classtag . for now.  just take as is.
+// this is a bit hacky, Shoud not be doing onlytry/catch here..
+def getOperator[C: ClassTag]: MMBinaryFunc = {
+
+  try {
+println("creating org.apache.mahout.viennacl.opencl.GPUMMul 
solver")
--- End diff --

I understand it is WIP. I imagined we should develop a concept of backends, 
rather than binding to a concrete solver like this. 

So backend = collection of available solvers for specific library. So for 
VCL, there's just one Backend (if we ignore the fact that multiple platforms 
may be present in either one or many VCL backend jars). 

Backends are enumerated and managed by SolverFactory. So SolverFactory has 
3 functions :
(1) probe and manage backends
(2) probe individual solver support in each of backeds and decide which one 
to instantiate and return
(3) manage available solvers. 

I imagine we won't do this initially, but we should leave enough 
flexibility in this class OOA so that we could insert rules allowing choosing 
among more than one active solver of the same type depending on additional 
factors.

Such things may include situations when e.g., multiple backends are 
available which do not share resources (i.e., gpu and host backends); or simple 
calibration estimates when for smaller problem sizes host backends are more 
efficient than device backends because they do not require problem repackaging 
and transfer between host and the device.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-11 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567553#comment-15567553
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user dlyubimov commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82933399
  
--- Diff: 
math-scala/src/main/scala/org/apache/mahout/math/scalabindings/SolverFactory.scala
 ---
@@ -0,0 +1,71 @@
+/**
+  * Licensed to the Apache Software Foundation (ASF) under one or more
+  * contributor license agreements.  See the NOTICE file distributed with
+  * this work for additional information regarding copyright ownership.
+  * The ASF licenses this file to You under the Apache License, Version 2.0
+  * (the "License"); you may not use this file except in compliance with
+  * the License.  You may obtain a copy of the License at
+  *
+  * http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+package org.apache.mahout.math.scalabindings
+
+import java.io.File
+
+import org.apache.mahout.logging._
+
+import scala.reflect.ClassTag
+import scala.reflect.runtime._
+import scala.reflect._
+
+
+class SolverFactory {
+
+  private final implicit val log = getLog(this.getClass)
+
+  // just temp for quick POC
+  val classMap: Map[String,String] =
+Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
--- End diff --

right, ok, but eventually it is probed and initialized on singleton init 
via (hopefully) a Backend's interface interrogations.
Typically, there are known (predefined) Backend set of classes + additional 
custom backends via a property. 

The backend concrete implementations do not generally have to be a 
dependency of this module (e.g., because there are no native javacpp codes for 
this platform), so generally there's no concrete dependency and hense class 
name cannot be used directly, only via reflection. 

On the other hand, in order to enable a custom (well-known) backend, it 
should be sufficient to drop in a version of such compiled for current platform 
for things to start moving around. 

Consider, for example, a blas Backend. then solvers there generally would 
not be available if (1) blas backend jar is not in the classpath; or (2) blas 
backend probed -- and reported back as unavailable (usually due to failure to 
bind to libBlas.so in the system).


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-11 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567475#comment-15567475
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82932654
  
--- Diff: 
math-scala/src/main/scala/org/apache/mahout/math/scalabindings/SolverFactory.scala
 ---
@@ -0,0 +1,71 @@
+/**
+  * Licensed to the Apache Software Foundation (ASF) under one or more
+  * contributor license agreements.  See the NOTICE file distributed with
+  * this work for additional information regarding copyright ownership.
+  * The ASF licenses this file to You under the Apache License, Version 2.0
+  * (the "License"); you may not use this file except in compliance with
+  * the License.  You may obtain a copy of the License at
+  *
+  * http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+package org.apache.mahout.math.scalabindings
+
+import java.io.File
+
+import org.apache.mahout.logging._
+
+import scala.reflect.ClassTag
+import scala.reflect.runtime._
+import scala.reflect._
+
+
+class SolverFactory {
+
+  private final implicit val log = getLog(this.getClass)
+
+  // just temp for quick POC
+  val classMap: Map[String,String] =
+Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
--- End diff --

not Sure that we can use classOf[..].getName, since the dependency is not 
guaranteed to be there.  Eg if we build with the default `mvn clean install`,  
GPUMMul will be no where on the classpath.  so `classOf[GPU]` would fail.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-11 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15565613#comment-15565613
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user smarthi commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82808490
  
--- Diff: 
spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala
 ---
@@ -162,7 +163,7 @@ class MahoutSparkILoop extends SparkILoop {
  _ __ ___   __ _| |__   ___  _   _| |_
 | '_ ` _ \ / _` | '_ \ / _ \| | | | __|
 | | | | | | (_| | | | | (_) | |_| | |_
-|_| |_| |_|\__,_|_| |_|\___/ \__,_|\__|  version 0.12.2
+|_| |_| |_|\__,_|_| |_|\___/ \__,_|\__|  version 0.13.0
 
--- End diff --

A question to the team for future releases - with Zeppelin now available, 
do we still want to maintain Mahout-Spark shell ?


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-11 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564602#comment-15564602
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82729074
  
--- Diff: 
spark/src/test/scala/org/apache/mahout/cf/SimilarityAnalysisSuite.scala ---
@@ -17,16 +17,14 @@
 
 package org.apache.mahout.cf
 
--- End diff --

Oh yeah- this cooccurance stuff- not sure how this made it in here...


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564584#comment-15564584
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82728207
  
--- Diff: 
viennacl-omp/src/main/scala/org/apache/mahout/viennacl/openmp/OMPMMul.scala ---
@@ -0,0 +1,448 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.mahout.viennacl.openmp
+
+import org.apache.mahout.logging._
+import org.apache.mahout.math
+import org.apache.mahout.math._
+import org.apache.mahout.math.flavor.{BackEnum, TraversingStructureEnum}
+import org.apache.mahout.math.function.Functions
+import org.apache.mahout.math.scalabindings.RLikeOps._
+import org.apache.mahout.math.scalabindings._
+import org.apache.mahout.viennacl.openmp.javacpp.Functions._
+import org.apache.mahout.viennacl.openmp.javacpp.LinalgFunctions._
+import org.apache.mahout.viennacl.openmp.javacpp.{CompressedMatrix, 
Context, DenseRowMatrix}
+
+import scala.collection.JavaConversions._
+
+object OMPMMul extends MMBinaryFunc {
+
+  private final implicit val log = getLog(OMPMMul.getClass)
+
+  override def apply(a: Matrix, b: Matrix, r: Option[Matrix]): Matrix = {
+
+require(a.ncol == b.nrow, "Incompatible matrix sizes in matrix 
multiplication.")
+
+val (af, bf) = (a.getFlavor, b.getFlavor)
+val backs = (af.getBacking, bf.getBacking)
+val sd = (af.getStructure, math.scalabindings.densityAnalysis(a), 
bf.getStructure, densityAnalysis(b))
+
+
+try {
+
+  val alg: MMulAlg = backs match {
+
+// Both operands are jvm memory backs.
+case (BackEnum.JVMMEM, BackEnum.JVMMEM) ⇒
+
+  sd match {
+
+// Multiplication cases by a diagonal matrix.
+case (TraversingStructureEnum.VECTORBACKED, _, 
TraversingStructureEnum.COLWISE, _)
+  if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagCW
+case (TraversingStructureEnum.VECTORBACKED, _, 
TraversingStructureEnum.SPARSECOLWISE, _)
+  if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagCW
+case (TraversingStructureEnum.VECTORBACKED, _, 
TraversingStructureEnum.ROWWISE, _)
+  if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagRW
+case (TraversingStructureEnum.VECTORBACKED, _, 
TraversingStructureEnum.SPARSEROWWISE, _)
+  if a.isInstanceOf[DiagonalMatrix] ⇒ jvmDiagRW
+
+case (TraversingStructureEnum.COLWISE, _, 
TraversingStructureEnum.VECTORBACKED, _)
+  if b.isInstanceOf[DiagonalMatrix] ⇒ jvmCWDiag
+case (TraversingStructureEnum.SPARSECOLWISE, _, 
TraversingStructureEnum.VECTORBACKED, _)
+  if b.isInstanceOf[DiagonalMatrix] ⇒ jvmCWDiag
+case (TraversingStructureEnum.ROWWISE, _, 
TraversingStructureEnum.VECTORBACKED, _)
+  if b.isInstanceOf[DiagonalMatrix] ⇒ jvmRWDiag
+case (TraversingStructureEnum.SPARSEROWWISE, _, 
TraversingStructureEnum.VECTORBACKED, _)
+  if b.isInstanceOf[DiagonalMatrix] ⇒ jvmRWDiag
+
+// Dense-dense cases
+case (TraversingStructureEnum.ROWWISE, true, 
TraversingStructureEnum.COLWISE, true) if a eq b.t ⇒ ompDRWAAt
+case (TraversingStructureEnum.ROWWISE, true, 
TraversingStructureEnum.COLWISE, true) if a.t eq b ⇒ ompDRWAAt
+case (TraversingStructureEnum.ROWWISE, true, 
TraversingStructureEnum.COLWISE, true) ⇒ ompRWCW
+case (TraversingStructureEnum.ROWWISE, true, 
TraversingStructureEnum.ROWWISE, true) ⇒ jvmRWRW
+case (TraversingStructureEnum.COLWISE, true, 
TraversingStructureEnum.COLWISE, true) ⇒ jvmCWCW
+case (TraversingStructureEnum.COLWISE, true, 
TraversingStructureEnum.ROWWISE, true) if a eq b.t

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564571#comment-15564571
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261
  
@sscdotopen thx for the review.  I did just rebase it on the current 
master, which changes do you see that are not related to VuennaCL?


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564574#comment-15564574
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82727932
  
--- Diff: math/src/main/java/org/apache/mahout/math/DenseVector.java ---
@@ -253,6 +253,13 @@ public void addAll(Vector v) {
 }
   }
 
+  /*
+   * return thr backing double array.
+   */
+  public double[] getBackingDataStructure(){
--- End diff --

this is actually never ended up needing.  Will revert this  change.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564515#comment-15564515
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82725812
  
--- Diff: examples/bin/SparseSparseDrmTimer.mscala ---
@@ -0,0 +1,37 @@
+
+
+def timeSparseDRMMMul(m: Int, n: Int, s: Int, para: Int, pctDense: Double 
= .20): Long = {
+
+ 
+
+  val drmA = drmParallelizeEmpty(m , s, para).mapBlock(){
+   case (keys,block:Matrix) =>
+ val R =  scala.util.Random
+ R.setSeed(1234L)
--- End diff --

If we use hardcoded seeds, we should probably define them as a constant 
somewhere


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564517#comment-15564517
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82725973
  
--- Diff: math/src/main/java/org/apache/mahout/math/DenseMatrix.java ---
@@ -65,6 +65,12 @@ public DenseMatrix(int rows, int columns) {
 super(rows, columns);
 this.values = new double[rows][columns];
   }
+  /*
--- End diff --

put the comment on a single line and leave a space after ()


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564518#comment-15564518
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82725859
  
--- Diff: examples/bin/testVcl.scala ---
@@ -0,0 +1,30 @@
+import org.apache.spark.SparkContext._
+import org.apache.spark.SparkContext
+import org.apache.spark.SparkConf
+
+
+
+object TestVCL {
+
+  def main(args: Array[String]) {
+run()
+  }
+
+  private def run() {
+val sparkConf = new SparkConf()
+sparkConf.setAppName("TestJob")
+sparkConf.set("spark.cores.max", "8")
+sparkConf.set("spark.storage.memoryFraction", "0.1")
+sparkConf.set("spark.shuffle.memoryFracton", "0.2")
+sparkConf.set("spark.executor.memory", "2g")
+
sparkConf.setJars(List("target/scala-2.10/spark-test-assembly-1.0.jar"))
+sparkConf.setMaster(s"spark://dev1.dev.pulse.io:7077")
+sparkConf.setSparkHome("/home/pulseio/spark/current")
--- End diff --

hardcoded path again


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564514#comment-15564514
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82725986
  
--- Diff: math/src/main/java/org/apache/mahout/math/DenseVector.java ---
@@ -253,6 +253,13 @@ public void addAll(Vector v) {
 }
   }
 
+  /*
+   * return thr backing double array.
+   */
+  public double[] getBackingDataStructure(){
--- End diff --

put the comment on a single line and leave a space after ()


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564516#comment-15564516
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82725845
  
--- Diff: examples/bin/testVcl.scala ---
@@ -0,0 +1,30 @@
+import org.apache.spark.SparkContext._
+import org.apache.spark.SparkContext
+import org.apache.spark.SparkConf
+
+
+
+object TestVCL {
+
+  def main(args: Array[String]) {
+run()
+  }
+
+  private def run() {
+val sparkConf = new SparkConf()
+sparkConf.setAppName("TestJob")
+sparkConf.set("spark.cores.max", "8")
+sparkConf.set("spark.storage.memoryFraction", "0.1")
+sparkConf.set("spark.shuffle.memoryFracton", "0.2")
+sparkConf.set("spark.executor.memory", "2g")
+
sparkConf.setJars(List("target/scala-2.10/spark-test-assembly-1.0.jar"))
+sparkConf.setMaster(s"spark://dev1.dev.pulse.io:7077")
--- End diff --

We should not have these hardcoded paths here probably


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564513#comment-15564513
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82726045
  
--- Diff: 
spark/src/main/scala/org/apache/mahout/sparkbindings/package.scala ---
@@ -75,7 +79,8 @@ package object sparkbindings {
 
   } else {
 // In local mode we don't care about jars, do we?
-sparkConf.setJars(customJars.toSeq)
+// yes adding jars always now since we are not including the 
artifacts
+ sparkConf.setJars(customJars.toSeq)
--- End diff --

one space too much here


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564519#comment-15564519
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user sscdotopen commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82725898
  
--- Diff: 
math-scala/src/main/scala/org/apache/mahout/math/scalabindings/SolverFactory.scala
 ---
@@ -0,0 +1,71 @@
+/**
+  * Licensed to the Apache Software Foundation (ASF) under one or more
+  * contributor license agreements.  See the NOTICE file distributed with
+  * this work for additional information regarding copyright ownership.
+  * The ASF licenses this file to You under the Apache License, Version 2.0
+  * (the "License"); you may not use this file except in compliance with
+  * the License.  You may obtain a copy of the License at
+  *
+  * http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+package org.apache.mahout.math.scalabindings
+
+import java.io.File
+
+import org.apache.mahout.logging._
+
+import scala.reflect.ClassTag
+import scala.reflect.runtime._
+import scala.reflect._
+
+
+class SolverFactory {
+
+  private final implicit val log = getLog(this.getClass)
+
+  // just temp for quick POC
+  val classMap: Map[String,String] =
+Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
--- End diff --

classOf[...].getName


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564487#comment-15564487
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82725138
  
--- Diff: LICENSE.txt ---
@@ -799,3 +799,32 @@ The following license applies to the H2O package
   identification within third-party archives.
 
Copyright 2012 0xdata, Inc
+
+
+The following applies to the ViennaCL library and files in the 
mahout-native-viennacl module
+
+
+   Copyright (c) 2010-2016 Institute for Microelectronics,
+   Institute for Analysis and Scientific 
Computing, TU Wien.
+   Portions of this software are copyright by UChicago Argonne, LLC.
+   Argonne National Laboratory, with facilities in the state of Illinois,
+   is owned by The United States Government, and operated by UChicago 
Argonne, LLC
+   under provision of a contract with the Department of Energy.
+
+   Permission is hereby granted, free of charge, to any person obtaining a 
copy
+   of this software and associated documentation files (the "Software"), 
to deal
+   in the Software without restriction, including without limitation the 
rights
+   to use, copy, modify, merge, publish, distribute, sublicense, and/or 
sell
+   copies of the Software, and to permit persons to whom the Software is
+   furnished to do so, subject to the following conditions:
+
+   The above copyright notice and this permission notice shall be included 
in
+   all copies or substantial portions of the Software.
+
+   THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS 
OR
+   IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+   FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL 
THE
+   AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+   LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 
FROM,
+   OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 
IN
+   THE SOFTWARE.
--- End diff --

yes definitly..


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564366#comment-15564366
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261
  
rebased on master


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562616#comment-15562616
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user smarthi commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82629295
  
--- Diff: 
math-scala/src/main/scala/org/apache/mahout/math/scalabindings/RLikeMatrixOps.scala
 ---
@@ -17,14 +17,22 @@
 package org.apache.mahout.math.scalabindings
 
 import org.apache.mahout.math.function.Functions
-import org.apache.mahout.math.{Vector, Matrix}
+import org.apache.mahout.math.{Matrix, Vector}
 import scala.collection.JavaConversions._
 import RLikeOps._
+import org.apache.mahout.math.scalabindings._
+
 
 class RLikeMatrixOps(m: Matrix) extends MatrixOps(m) {
 
   /** Structure-optimized mmul */
-  def %*%(that: Matrix) = MMul(m, that, None)
+
+  implicit var solverOperator: opMMulSolver =_
--- End diff --

leave a space after '='


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562619#comment-15562619
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user smarthi commented on a diff in the pull request:

https://github.com/apache/mahout/pull/261#discussion_r82629450
  
--- Diff: pom.xml ---
@@ -803,34 +810,36 @@
 buildtools
 math
 hdfs
-mr
-integration
-examples
+
+
+
 distribution
 math-scala
 spark
 spark-shell
 flink
 h2o
+
+
   
 
   
+
 
-  hadoop1
-  
-hadoop1
-1.2.1
-  
-  
-
-  
-org.apache.hadoop
-hadoop-core
-${hadoop.version}
-  
-
-  
+  viennacl
--- End diff --

hadoop1 profile has been removed, rebase ur code and update the PR


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15560908#comment-15560908
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


GitHub user andrewpalumbo opened a pull request:

https://github.com/apache/mahout/pull/261

MAHOUT-1885 [WIP][FOR COMMENT]: Inital implementation of VCL bindings for 
mahout math.

This is an initial set of bindings for VCL to mahout matrices.  Currently 
supports `DenseRowMatrix` and `SparseRowMatrix` for both GPU and multi-threaded 
CPU.

Note that there are two new modules:  `vienniacl` and `viennacl-omp`  there 
are new profiles to activate these: `-Pvienniacl` activates both and  
`-Pviennacl-omp` activates only the OpenMP version.

The default build should not require any new installs.

If activating either version you';; need to install vienniacl 1.7.2+ 

On ubuntu 16.04 this is simply:
```
sudo apt-get install libviennacl-dev
```
if running on GPU you'll also need, you'll need to have OpenCL 1.2+, which 
is likely installed with your video driver.

Hadoop2 is still in a profile which needs to be removed, so the command to 
install for GPU and CPU is:
```
mvn clean install -Pviennacl -Phadoop2 -DskipTests && mvn test
```
for OpenMP only  (mac users) :
```
mvn clean install -Pviennacl -Phadoop2 -DskipTests && mvn test
```

The output is still very verbose and will let you know each time a `%*%` 
operatoion is made which device it is made on.

A few todos:

1.  Not sure that we need two separate modules for omp and gpu.  There may 
be a way to handle this with environment variables. 
2. Currently a new object is being created each timw `%*%` is called.  it 
would be best to create these once and then cache them for subsequent calls.
3.  Currently using try/catch for `XXXMMul` creation, really bad style.
4.  GPU Vector are not working and throw exceptions when trying to read 
data out of them
   - Vector this we could implement at least one native linear solver. 
(most viennacl linear solvers require boost however this is slated for removal 
on the next release).
5.  Have been noticing intermittent crashes on GPU.
6.  Tests with CLI drivers are currently commented out.  Was getting odd 
non-numeric failures with them.  must be a config issue.
7.  Currently each operation can only be run on one GPU /node.

I'd like to get this committed relatively quickly and then go back and fine 
tune.

Any feedback is much appreciated. 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewpalumbo/mahout viennacl-opmmul-a

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/mahout/pull/261.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #261


commit dfc93684a3d17a0331504a45b68ad2dfb9520f5c
Author: Andrew Palumbo 
Date:   2016-08-10T17:48:51Z

vienniacl back POC for mahout dense and sparse matrix multiplication.  
Compiling and testing as is with a couple of tests commented out.  Having 
issues currently with Dense %*% Sparse, and reading data from GPU after Matrix 
%*% Vector multiplication.

commit 5d96a17bb828b2c2a887ddd48f9eede4e13eef92
Author: Andrew Palumbo 
Date:   2016-08-16T20:32:47Z

fix imports for Sparse/Dense MMul

commit c2a548e239d31fee56428e7a02c8a0f76de01915
Author: Andrew Palumbo 
Date:   2016-08-16T20:51:24Z

Add functions to LinalgFunctions.java for direct matrix prod expressions, 
remove import.  Uncomment missed OpenCL test

commit 8de0fd03821a2bd08bd469177214567c6ae60b66
Author: Andrew Palumbo 
Date:   2016-08-31T16:46:19Z

allow spark build in module.  Add a linux-haswell properties file- this may 
need a property in viennacl/pom.xml to be used. As well will need one for mac.  
This is for the distribution artifact which may not be built by a 'native' 
machine.  Currently leaving linux-x86_64 to use -march=native but as this will 
likely need to be changed as suneel does releases from a mac and I (AP) use an 
amdfam10 so we should have a general linux-x86_64 properties file.

commit 1525dbe389aebd3ea1244ec4468cff08fb683cda
Author: Andrew Palumbo 
Date:   2016-08-31T20:19:43Z

[EXPIRMENTAL BRANCH] Split OpenCL and OpenMP modules up.  Next steps: 
implement MMulGPU an MMulOMP and SolverFactory per doc, test on AWS GPU Cluster.

commit a6a81f8ced8f24dfbafc655e46822a9ddfd2c90a
Author: Andrew Palumbo 
Date:   2016-09-07T17:21:49Z

[EXPIRMENTAL BRANCH] Hacked everything together in hopes of having a 
running branch to test on AWS byt Wed 9/7

45 matches

Mail list logo