date:20161017

Re: Trying out PR 261

2016-10-17 Thread Andrew Musselman

"Works" doing that pom hack but agreed need to make it work without it.

On Mon, Oct 17, 2016 at 10:41 AM, Andrew Musselman <
andrew.mussel...@gmail.com> wrote:

> No this was with the -omp profile; I'm going to try out his hack now for
> non-omp.
>
> On Mon, Oct 17, 2016 at 10:15 AM, Andrew Palumbo 
> wrote:
>
>> That is weird.. did u tried it with Trevor's fix?
>>
>>
>>
>> Sent from my Verizon Wireless 4G LTE smartphone
>>
>>
>>  Original message 
>> From: Andrew Musselman 
>> Date: 10/17/2016 10:11 AM (GMT-08:00)
>> To: dev@mahout.apache.org
>> Subject: Re: Trying out PR 261
>>
>> I got the error before setting Xmx however.
>>
>> On Mon, Oct 17, 2016 at 10:03 AM, Andrew Palumbo 
>> wrote:
>>
>> > I think that the memory should be set to 3 or  4G in the tests in
>> > pom.xml.. maybe you're o erring that setting?
>> >
>> >
>> >
>> > Sent from my Verizon Wireless 4G LTE smartphone
>> >
>> >
>> >  Original message 
>> > From: Andrew Musselman 
>> > Date: 10/17/2016 9:47 AM (GMT-08:00)
>> > To: dev@mahout.apache.org
>> > Subject: Re: Trying out PR 261
>> >
>> > Using the viennacl-omp profile I'm getting an out of memory error even
>> with
>> > Xmx set in MAHOUT_OPTS to 1g:
>> >
>> > - mmul microbenchmark
>> >   + Mahout multiplication time: 8920 ms.
>> >   + ViennaCL/cpu/OpenMP multiplication time: 616 ms.
>> > - trans
>> > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
>> > [WARN] Unable to create class GPUMMul: attempting OpenMP version
>> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
>> > [INFO] Unable to create class OMPMMul: falling back to java version
>> > *** RUN ABORTED ***
>> >   java.lang.OutOfMemoryError: Java heap space
>> >   at
>> > it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.rehash(
>> > Int2DoubleOpenHashMap.java:1059)
>> >   at
>> > it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.insert(
>> > Int2DoubleOpenHashMap.java:295)
>> >   at
>> > it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.put(
>> > Int2DoubleOpenHashMap.java:301)
>> >   at
>> > org.apache.mahout.math.RandomAccessSparseVector.setQuick(
>> > RandomAccessSparseVector.java:130)
>> >   at
>> > org.apache.mahout.math.SparseRowMatrix.setQuick(SparseRowMat
>> rix.java:105)
>> >   at org.apache.mahout.math.AbstractMatrix.assign(AbstractMatrix.
>> java:256)
>> >   at
>> > org.apache.mahout.math.scalabindings.MatrixOps.$
>> > colon$eq(MatrixOps.scala:192)
>> >   at
>> > org.apache.mahout.math.scalabindings.MatrixOps.cloned(
>> MatrixOps.scala:260)
>> >   at
>> > org.apache.mahout.math.scalabindings.MatrixOps.$minus(
>> MatrixOps.scala:66)
>> >   at
>> > org.apache.mahout.viennacl.openmp.ViennaCLSuiteOMP$$
>> > anonfun$4.apply$mcV$sp(ViennaCLSuiteOMP.scala:152)
>> >   ...
>> > [INFO]
>> > 
>> 
>> > [INFO] Reactor Summary:
>> > [INFO]
>> > [INFO] Mahout Build Tools . SUCCESS [
>> >  1.131 s]
>> > [INFO] Apache Mahout .. SUCCESS [
>> >  0.023 s]
>> > [INFO] Mahout Math  SUCCESS [
>> > 31.439 s]
>> > [INFO] Mahout HDFS  SUCCESS [
>> >  2.151 s]
>> > [INFO] Mahout Math Scala bindings . SUCCESS
>> [02:30
>> > min]
>> > [INFO] Mahout Spark bindings .. SUCCESS [
>> > 43.884 s]
>> > [INFO] Mahout Flink bindings .. SUCCESS
>> [04:13
>> > min]
>> > [INFO] Mahout Spark bindings shell  SUCCESS [
>> >  4.071 s]
>> > [INFO] Mahout Release Package . SUCCESS [
>> >  1.311 s]
>> > [INFO] Mahout H2O backend . SUCCESS
>> [02:50
>> > min]
>> > [INFO] Mahout Native VienniaCL OpenMP Bindings  FAILURE [
>> > 36.106 s]
>> >
>> > On Sun, Oct 16, 2016 at 9:43 PM, Trevor Grant > >
>> > wrote:
>> >
>> > > NOTE:
>> > >
>> > > The last solution creates a cyclical dependency-
>> > >
>> > > The way I made it work was happen-stance.
>> > >
>> > > 1)
>> > > mvn clean install -Pviennacl -Phadoop2 -DskipTests
>> > >
>> > > 2)
>> > > update the pom as described previously
>> > >
>> > > 3)
>> > > cd math-scala
>> > >
>> > > 4)
>> > > mvn test
>> > >
>> > > that will make it work- hacky but POC.
>> > >
>> > > Next how to do this 'right'.
>> > >
>> > > tg
>> > >
>> > >
>> > >
>> > > Trevor Grant
>> > > Data Scientist
>> > > https://github.com/rawkintrevo
>> > > http://stackexchange.com/users/3002022/rawkintrevo
>> > > http://trevorgrant.org
>> > >
>> > > *"Fortunate is he, who is able to know the causes of things."
>> -Virgil*
>> > >
>> > >
>> > > On Sun, Oct 16, 2016 at 11:29 PM, Trevor Grant <
>> trevor.d.gr...@gmail.com
>> > >
>> > > wrote:
>> > >
>> > > > Maybe not the long term fix, but

Re: Trying out PR 261

2016-10-17 Thread Andrew Musselman

No this was with the -omp profile; I'm going to try out his hack now for
non-omp.

On Mon, Oct 17, 2016 at 10:15 AM, Andrew Palumbo  wrote:

> That is weird.. did u tried it with Trevor's fix?
>
>
>
> Sent from my Verizon Wireless 4G LTE smartphone
>
>
>  Original message 
> From: Andrew Musselman 
> Date: 10/17/2016 10:11 AM (GMT-08:00)
> To: dev@mahout.apache.org
> Subject: Re: Trying out PR 261
>
> I got the error before setting Xmx however.
>
> On Mon, Oct 17, 2016 at 10:03 AM, Andrew Palumbo 
> wrote:
>
> > I think that the memory should be set to 3 or  4G in the tests in
> > pom.xml.. maybe you're o erring that setting?
> >
> >
> >
> > Sent from my Verizon Wireless 4G LTE smartphone
> >
> >
> >  Original message 
> > From: Andrew Musselman 
> > Date: 10/17/2016 9:47 AM (GMT-08:00)
> > To: dev@mahout.apache.org
> > Subject: Re: Trying out PR 261
> >
> > Using the viennacl-omp profile I'm getting an out of memory error even
> with
> > Xmx set in MAHOUT_OPTS to 1g:
> >
> > - mmul microbenchmark
> >   + Mahout multiplication time: 8920 ms.
> >   + ViennaCL/cpu/OpenMP multiplication time: 616 ms.
> > - trans
> > [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> > [WARN] Unable to create class GPUMMul: attempting OpenMP version
> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> > [INFO] Unable to create class OMPMMul: falling back to java version
> > *** RUN ABORTED ***
> >   java.lang.OutOfMemoryError: Java heap space
> >   at
> > it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.rehash(
> > Int2DoubleOpenHashMap.java:1059)
> >   at
> > it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.insert(
> > Int2DoubleOpenHashMap.java:295)
> >   at
> > it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.put(
> > Int2DoubleOpenHashMap.java:301)
> >   at
> > org.apache.mahout.math.RandomAccessSparseVector.setQuick(
> > RandomAccessSparseVector.java:130)
> >   at
> > org.apache.mahout.math.SparseRowMatrix.setQuick(
> SparseRowMatrix.java:105)
> >   at org.apache.mahout.math.AbstractMatrix.assign(
> AbstractMatrix.java:256)
> >   at
> > org.apache.mahout.math.scalabindings.MatrixOps.$
> > colon$eq(MatrixOps.scala:192)
> >   at
> > org.apache.mahout.math.scalabindings.MatrixOps.
> cloned(MatrixOps.scala:260)
> >   at
> > org.apache.mahout.math.scalabindings.MatrixOps.$
> minus(MatrixOps.scala:66)
> >   at
> > org.apache.mahout.viennacl.openmp.ViennaCLSuiteOMP$$
> > anonfun$4.apply$mcV$sp(ViennaCLSuiteOMP.scala:152)
> >   ...
> > [INFO]
> > 
> > [INFO] Reactor Summary:
> > [INFO]
> > [INFO] Mahout Build Tools . SUCCESS [
> >  1.131 s]
> > [INFO] Apache Mahout .. SUCCESS [
> >  0.023 s]
> > [INFO] Mahout Math  SUCCESS [
> > 31.439 s]
> > [INFO] Mahout HDFS  SUCCESS [
> >  2.151 s]
> > [INFO] Mahout Math Scala bindings . SUCCESS
> [02:30
> > min]
> > [INFO] Mahout Spark bindings .. SUCCESS [
> > 43.884 s]
> > [INFO] Mahout Flink bindings .. SUCCESS
> [04:13
> > min]
> > [INFO] Mahout Spark bindings shell  SUCCESS [
> >  4.071 s]
> > [INFO] Mahout Release Package . SUCCESS [
> >  1.311 s]
> > [INFO] Mahout H2O backend . SUCCESS
> [02:50
> > min]
> > [INFO] Mahout Native VienniaCL OpenMP Bindings  FAILURE [
> > 36.106 s]
> >
> > On Sun, Oct 16, 2016 at 9:43 PM, Trevor Grant 
> > wrote:
> >
> > > NOTE:
> > >
> > > The last solution creates a cyclical dependency-
> > >
> > > The way I made it work was happen-stance.
> > >
> > > 1)
> > > mvn clean install -Pviennacl -Phadoop2 -DskipTests
> > >
> > > 2)
> > > update the pom as described previously
> > >
> > > 3)
> > > cd math-scala
> > >
> > > 4)
> > > mvn test
> > >
> > > that will make it work- hacky but POC.
> > >
> > > Next how to do this 'right'.
> > >
> > > tg
> > >
> > >
> > >
> > > Trevor Grant
> > > Data Scientist
> > > https://github.com/rawkintrevo
> > > http://stackexchange.com/users/3002022/rawkintrevo
> > > http://trevorgrant.org
> > >
> > > *"Fortunate is he, who is able to know the causes of things."  -Virgil*
> > >
> > >
> > > On Sun, Oct 16, 2016 at 11:29 PM, Trevor Grant <
> trevor.d.gr...@gmail.com
> > >
> > > wrote:
> > >
> > > > Maybe not the long term fix, but adding
> > > >
> > > > 
> > > >  org.apache.mahout
> > > >  mahout-native-viennacl_${scala.compat.
> > > > version}
> > > >  test
> > > >
> > > >
> > > > to math-scala/pom.xml seems to fix the afore mentioned errors
> > > >
> > > >
> > > > Trevor Grant
> > > > Data Scientist
> > > > https://github.com/rawkintrevo

RE: Trying out PR 261

2016-10-17 Thread Andrew Palumbo

That is weird.. did u tried it with Trevor's fix?



Sent from my Verizon Wireless 4G LTE smartphone


 Original message 
From: Andrew Musselman 
Date: 10/17/2016 10:11 AM (GMT-08:00)
To: dev@mahout.apache.org
Subject: Re: Trying out PR 261

I got the error before setting Xmx however.

On Mon, Oct 17, 2016 at 10:03 AM, Andrew Palumbo  wrote:

> I think that the memory should be set to 3 or  4G in the tests in
> pom.xml.. maybe you're o erring that setting?
>
>
>
> Sent from my Verizon Wireless 4G LTE smartphone
>
>
>  Original message 
> From: Andrew Musselman 
> Date: 10/17/2016 9:47 AM (GMT-08:00)
> To: dev@mahout.apache.org
> Subject: Re: Trying out PR 261
>
> Using the viennacl-omp profile I'm getting an out of memory error even with
> Xmx set in MAHOUT_OPTS to 1g:
>
> - mmul microbenchmark
>   + Mahout multiplication time: 8920 ms.
>   + ViennaCL/cpu/OpenMP multiplication time: 616 ms.
> - trans
> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> [WARN] Unable to create class GPUMMul: attempting OpenMP version
> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> [INFO] Unable to create class OMPMMul: falling back to java version
> *** RUN ABORTED ***
>   java.lang.OutOfMemoryError: Java heap space
>   at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.rehash(
> Int2DoubleOpenHashMap.java:1059)
>   at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.insert(
> Int2DoubleOpenHashMap.java:295)
>   at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.put(
> Int2DoubleOpenHashMap.java:301)
>   at
> org.apache.mahout.math.RandomAccessSparseVector.setQuick(
> RandomAccessSparseVector.java:130)
>   at
> org.apache.mahout.math.SparseRowMatrix.setQuick(SparseRowMatrix.java:105)
>   at org.apache.mahout.math.AbstractMatrix.assign(AbstractMatrix.java:256)
>   at
> org.apache.mahout.math.scalabindings.MatrixOps.$
> colon$eq(MatrixOps.scala:192)
>   at
> org.apache.mahout.math.scalabindings.MatrixOps.cloned(MatrixOps.scala:260)
>   at
> org.apache.mahout.math.scalabindings.MatrixOps.$minus(MatrixOps.scala:66)
>   at
> org.apache.mahout.viennacl.openmp.ViennaCLSuiteOMP$$
> anonfun$4.apply$mcV$sp(ViennaCLSuiteOMP.scala:152)
>   ...
> [INFO]
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Mahout Build Tools . SUCCESS [
>  1.131 s]
> [INFO] Apache Mahout .. SUCCESS [
>  0.023 s]
> [INFO] Mahout Math  SUCCESS [
> 31.439 s]
> [INFO] Mahout HDFS  SUCCESS [
>  2.151 s]
> [INFO] Mahout Math Scala bindings . SUCCESS [02:30
> min]
> [INFO] Mahout Spark bindings .. SUCCESS [
> 43.884 s]
> [INFO] Mahout Flink bindings .. SUCCESS [04:13
> min]
> [INFO] Mahout Spark bindings shell  SUCCESS [
>  4.071 s]
> [INFO] Mahout Release Package . SUCCESS [
>  1.311 s]
> [INFO] Mahout H2O backend . SUCCESS [02:50
> min]
> [INFO] Mahout Native VienniaCL OpenMP Bindings  FAILURE [
> 36.106 s]
>
> On Sun, Oct 16, 2016 at 9:43 PM, Trevor Grant 
> wrote:
>
> > NOTE:
> >
> > The last solution creates a cyclical dependency-
> >
> > The way I made it work was happen-stance.
> >
> > 1)
> > mvn clean install -Pviennacl -Phadoop2 -DskipTests
> >
> > 2)
> > update the pom as described previously
> >
> > 3)
> > cd math-scala
> >
> > 4)
> > mvn test
> >
> > that will make it work- hacky but POC.
> >
> > Next how to do this 'right'.
> >
> > tg
> >
> >
> >
> > Trevor Grant
> > Data Scientist
> > https://github.com/rawkintrevo
> > http://stackexchange.com/users/3002022/rawkintrevo
> > http://trevorgrant.org
> >
> > *"Fortunate is he, who is able to know the causes of things."  -Virgil*
> >
> >
> > On Sun, Oct 16, 2016 at 11:29 PM, Trevor Grant  >
> > wrote:
> >
> > > Maybe not the long term fix, but adding
> > >
> > > 
> > >  org.apache.mahout
> > >  mahout-native-viennacl_${scala.compat.
> > > version}
> > >  test
> > >
> > >
> > > to math-scala/pom.xml seems to fix the afore mentioned errors
> > >
> > >
> > > Trevor Grant
> > > Data Scientist
> > > https://github.com/rawkintrevo
> > > http://stackexchange.com/users/3002022/rawkintrevo
> > > http://trevorgrant.org
> > >
> > > *"Fortunate is he, who is able to know the causes of things."  -Virgil*
> > >
> > >
> > > On Sun, Oct 16, 2016 at 10:23 PM, Andrew Palumbo 
> > > wrote:
> > >
> > >> Thanks Trevor and Andrew- I think that both of your environments are
> > >> fine.. (Ubuntu 16.04 was just to recommended because ViennaCL 1.7.1 is
> > >> easily installed with sudo-apt-get install.  As

Re: Trying out PR 261

2016-10-17 Thread Andrew Musselman

I got the error before setting Xmx however.

On Mon, Oct 17, 2016 at 10:03 AM, Andrew Palumbo  wrote:

> I think that the memory should be set to 3 or  4G in the tests in
> pom.xml.. maybe you're o erring that setting?
>
>
>
> Sent from my Verizon Wireless 4G LTE smartphone
>
>
>  Original message 
> From: Andrew Musselman 
> Date: 10/17/2016 9:47 AM (GMT-08:00)
> To: dev@mahout.apache.org
> Subject: Re: Trying out PR 261
>
> Using the viennacl-omp profile I'm getting an out of memory error even with
> Xmx set in MAHOUT_OPTS to 1g:
>
> - mmul microbenchmark
>   + Mahout multiplication time: 8920 ms.
>   + ViennaCL/cpu/OpenMP multiplication time: 616 ms.
> - trans
> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> [WARN] Unable to create class GPUMMul: attempting OpenMP version
> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> [INFO] Unable to create class OMPMMul: falling back to java version
> *** RUN ABORTED ***
>   java.lang.OutOfMemoryError: Java heap space
>   at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.rehash(
> Int2DoubleOpenHashMap.java:1059)
>   at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.insert(
> Int2DoubleOpenHashMap.java:295)
>   at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.put(
> Int2DoubleOpenHashMap.java:301)
>   at
> org.apache.mahout.math.RandomAccessSparseVector.setQuick(
> RandomAccessSparseVector.java:130)
>   at
> org.apache.mahout.math.SparseRowMatrix.setQuick(SparseRowMatrix.java:105)
>   at org.apache.mahout.math.AbstractMatrix.assign(AbstractMatrix.java:256)
>   at
> org.apache.mahout.math.scalabindings.MatrixOps.$
> colon$eq(MatrixOps.scala:192)
>   at
> org.apache.mahout.math.scalabindings.MatrixOps.cloned(MatrixOps.scala:260)
>   at
> org.apache.mahout.math.scalabindings.MatrixOps.$minus(MatrixOps.scala:66)
>   at
> org.apache.mahout.viennacl.openmp.ViennaCLSuiteOMP$$
> anonfun$4.apply$mcV$sp(ViennaCLSuiteOMP.scala:152)
>   ...
> [INFO]
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Mahout Build Tools . SUCCESS [
>  1.131 s]
> [INFO] Apache Mahout .. SUCCESS [
>  0.023 s]
> [INFO] Mahout Math  SUCCESS [
> 31.439 s]
> [INFO] Mahout HDFS  SUCCESS [
>  2.151 s]
> [INFO] Mahout Math Scala bindings . SUCCESS [02:30
> min]
> [INFO] Mahout Spark bindings .. SUCCESS [
> 43.884 s]
> [INFO] Mahout Flink bindings .. SUCCESS [04:13
> min]
> [INFO] Mahout Spark bindings shell  SUCCESS [
>  4.071 s]
> [INFO] Mahout Release Package . SUCCESS [
>  1.311 s]
> [INFO] Mahout H2O backend . SUCCESS [02:50
> min]
> [INFO] Mahout Native VienniaCL OpenMP Bindings  FAILURE [
> 36.106 s]
>
> On Sun, Oct 16, 2016 at 9:43 PM, Trevor Grant 
> wrote:
>
> > NOTE:
> >
> > The last solution creates a cyclical dependency-
> >
> > The way I made it work was happen-stance.
> >
> > 1)
> > mvn clean install -Pviennacl -Phadoop2 -DskipTests
> >
> > 2)
> > update the pom as described previously
> >
> > 3)
> > cd math-scala
> >
> > 4)
> > mvn test
> >
> > that will make it work- hacky but POC.
> >
> > Next how to do this 'right'.
> >
> > tg
> >
> >
> >
> > Trevor Grant
> > Data Scientist
> > https://github.com/rawkintrevo
> > http://stackexchange.com/users/3002022/rawkintrevo
> > http://trevorgrant.org
> >
> > *"Fortunate is he, who is able to know the causes of things."  -Virgil*
> >
> >
> > On Sun, Oct 16, 2016 at 11:29 PM, Trevor Grant  >
> > wrote:
> >
> > > Maybe not the long term fix, but adding
> > >
> > > 
> > >  org.apache.mahout
> > >  mahout-native-viennacl_${scala.compat.
> > > version}
> > >  test
> > >
> > >
> > > to math-scala/pom.xml seems to fix the afore mentioned errors
> > >
> > >
> > > Trevor Grant
> > > Data Scientist
> > > https://github.com/rawkintrevo
> > > http://stackexchange.com/users/3002022/rawkintrevo
> > > http://trevorgrant.org
> > >
> > > *"Fortunate is he, who is able to know the causes of things."  -Virgil*
> > >
> > >
> > > On Sun, Oct 16, 2016 at 10:23 PM, Andrew Palumbo 
> > > wrote:
> > >
> > >> Thanks Trevor and Andrew- I think that both of your environments are
> > >> fine.. (Ubuntu 16.04 was just to recommended because ViennaCL 1.7.1 is
> > >> easily installed with sudo-apt-get install.  As logn as you have 1.7.1
> > on
> > >> the correct path, you should be fine.
> > >>
> > >>
> > >> I'm getting these errors now also, and by viewing the stack trace, I
> can
> > >> see that the mahout_native_vienniacl*.jars did not make it onto the
> > >> classpath some how:
> > >>
> > >>

RE: Trying out PR 261

2016-10-17 Thread Andrew Palumbo

I think that the memory should be set to 3 or  4G in the tests in pom.xml.. 
maybe you're o erring that setting?



Sent from my Verizon Wireless 4G LTE smartphone


 Original message 
From: Andrew Musselman 
Date: 10/17/2016 9:47 AM (GMT-08:00)
To: dev@mahout.apache.org
Subject: Re: Trying out PR 261

Using the viennacl-omp profile I'm getting an out of memory error even with
Xmx set in MAHOUT_OPTS to 1g:

- mmul microbenchmark
  + Mahout multiplication time: 8920 ms.
  + ViennaCL/cpu/OpenMP multiplication time: 616 ms.
- trans
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[WARN] Unable to create class GPUMMul: attempting OpenMP version
[INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
[INFO] Unable to create class OMPMMul: falling back to java version
*** RUN ABORTED ***
  java.lang.OutOfMemoryError: Java heap space
  at
it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.rehash(Int2DoubleOpenHashMap.java:1059)
  at
it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.insert(Int2DoubleOpenHashMap.java:295)
  at
it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.put(Int2DoubleOpenHashMap.java:301)
  at
org.apache.mahout.math.RandomAccessSparseVector.setQuick(RandomAccessSparseVector.java:130)
  at
org.apache.mahout.math.SparseRowMatrix.setQuick(SparseRowMatrix.java:105)
  at org.apache.mahout.math.AbstractMatrix.assign(AbstractMatrix.java:256)
  at
org.apache.mahout.math.scalabindings.MatrixOps.$colon$eq(MatrixOps.scala:192)
  at
org.apache.mahout.math.scalabindings.MatrixOps.cloned(MatrixOps.scala:260)
  at
org.apache.mahout.math.scalabindings.MatrixOps.$minus(MatrixOps.scala:66)
  at
org.apache.mahout.viennacl.openmp.ViennaCLSuiteOMP$$anonfun$4.apply$mcV$sp(ViennaCLSuiteOMP.scala:152)
  ...
[INFO]

[INFO] Reactor Summary:
[INFO]
[INFO] Mahout Build Tools . SUCCESS [
 1.131 s]
[INFO] Apache Mahout .. SUCCESS [
 0.023 s]
[INFO] Mahout Math  SUCCESS [
31.439 s]
[INFO] Mahout HDFS  SUCCESS [
 2.151 s]
[INFO] Mahout Math Scala bindings . SUCCESS [02:30
min]
[INFO] Mahout Spark bindings .. SUCCESS [
43.884 s]
[INFO] Mahout Flink bindings .. SUCCESS [04:13
min]
[INFO] Mahout Spark bindings shell  SUCCESS [
 4.071 s]
[INFO] Mahout Release Package . SUCCESS [
 1.311 s]
[INFO] Mahout H2O backend . SUCCESS [02:50
min]
[INFO] Mahout Native VienniaCL OpenMP Bindings  FAILURE [
36.106 s]

On Sun, Oct 16, 2016 at 9:43 PM, Trevor Grant 
wrote:

> NOTE:
>
> The last solution creates a cyclical dependency-
>
> The way I made it work was happen-stance.
>
> 1)
> mvn clean install -Pviennacl -Phadoop2 -DskipTests
>
> 2)
> update the pom as described previously
>
> 3)
> cd math-scala
>
> 4)
> mvn test
>
> that will make it work- hacky but POC.
>
> Next how to do this 'right'.
>
> tg
>
>
>
> Trevor Grant
> Data Scientist
> https://github.com/rawkintrevo
> http://stackexchange.com/users/3002022/rawkintrevo
> http://trevorgrant.org
>
> *"Fortunate is he, who is able to know the causes of things."  -Virgil*
>
>
> On Sun, Oct 16, 2016 at 11:29 PM, Trevor Grant 
> wrote:
>
> > Maybe not the long term fix, but adding
> >
> > 
> >  org.apache.mahout
> >  mahout-native-viennacl_${scala.compat.
> > version}
> >  test
> >
> >
> > to math-scala/pom.xml seems to fix the afore mentioned errors
> >
> >
> > Trevor Grant
> > Data Scientist
> > https://github.com/rawkintrevo
> > http://stackexchange.com/users/3002022/rawkintrevo
> > http://trevorgrant.org
> >
> > *"Fortunate is he, who is able to know the causes of things."  -Virgil*
> >
> >
> > On Sun, Oct 16, 2016 at 10:23 PM, Andrew Palumbo 
> > wrote:
> >
> >> Thanks Trevor and Andrew- I think that both of your environments are
> >> fine.. (Ubuntu 16.04 was just to recommended because ViennaCL 1.7.1 is
> >> easily installed with sudo-apt-get install.  As logn as you have 1.7.1
> on
> >> the correct path, you should be fine.
> >>
> >>
> >> I'm getting these errors now also, and by viewing the stack trace, I can
> >> see that the mahout_native_vienniacl*.jars did not make it onto the
> >> classpath some how:
> >>
> >>
> >> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> >> [WARN] Unable to create class GPUMMul: attempting OpenMP version
> >> java.lang.ClassNotFoundException: org.apache.mahout.viennacl.ope
> >> ncl.GPUMMul$
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> >> at

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582758#comment-15582758
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261
  
I don't think  that cyclical referencing is an issue.  There are other ways 
to get the class onto the path.  They just got dropped somehow when I was 
creating/reason the pr.  For now though adding the dependency at test time 
works for me just to make sure everything's working.



Sent from my Verizon Wireless 4G LTE smartphone


 Original message 
From: Trevor Grant 
Date: 10/16/2016 9:54 PM (GMT-08:00)
To: apache/mahout 
Cc: Andrew Palumbo , Author 
Subject: Re: [apache/mahout] MAHOUT-1885 [WIP][FOR COMMENT]: Inital 
implementation of VCL bindings for mahout math. (#261)


@rawkintrevo commented on this pull request.



In 
math-scala/src/main/scala/org/apache/mahout/math/scalabindings/SolverFactory.scala:

> +  // just temp for quick POC
+  val classMap: Map[String,String] =
+Map(("GPUMMul"->"org.apache.mahout.viennacl.opencl.GPUMMul"),
+("OMPMMul"->"org.apache.mahout.viennacl.openmp.OMPMMul"))
+}
+object SolverFactory extends SolverFactory {
+
+// default is JVM
+var clazz: MMBinaryFunc = MMul
+
+// eventually match on implicit Classtag . for now.  just take as is.
+// this is a bit hacky, Shoud not be doing onlytry/catch here..
+def getOperator[C: ClassTag]: MMBinaryFunc = {
+
+  try {
+println("creating org.apache.mahout.viennacl.opencl.GPUMMul 
solver")


given the cyclical reference issues, maybe a solution here should come 
sooner rather than later (e.g. solving this, would also 'fix' the cyclical 
reference problem?)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub, or mute the 
thread.



> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582746#comment-15582746
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261
  
I'd really opened this PR so that we could discuss architecture and the 
correct entry points for the SolverFactory, hence the ridiculously verbose and 
confusing debugging output.  I'd like to get the arch right before we commit.  
At its core it works as is, but there are alot of changes that need to be made.

Eg.  We may be able to get away with only one Viennacl module by setting an 
environment variable and checking for it in Context.scala if it exists we 
"#define VIENNACL_WITH_OPENCL" which will make the module OpenMP only.  The 
Problem With This Is That We Can't Test Gpu in this module.. without checking 
for the environment var.

The other big thing is that I don't know if the placement if the 
SolverFactory as it is is correct.. we may want to examine this, as well as 
cache a XXXMMUL Of both types the first time this class is hit so that we are 
not dynamically loading a class each time %*% is hit.

These are a couple of things that I'd like to get figured out before 
committing.

We've already identified a few pain points in the testing that everybody 
did over the weekend.  So let's get it right before committing.


Sent from my Verizon Wireless 4G LTE smartphone


 Original message 
From: Dmitriy Lyubimov 
Date: 10/17/2016 8:51 AM (GMT-08:00)
To: apache/mahout 
Cc: Andrew Palumbo , Author 
Subject: Re: [apache/mahout] MAHOUT-1885 [WIP][FOR COMMENT]: Inital 
implementation of VCL bindings for mahout math. (#261)


IMO this needs more work and committing this is premature at this point due 
to architecture considerations. Not only it misses some things, but also IMO 
has things that should not be done the way they currently are.

-
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub, or 
mute the 
thread.



> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582747#comment-15582747
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user andrewpalumbo commented on the issue:

https://github.com/apache/mahout/pull/261
  
+1



Sent from my Verizon Wireless 4G LTE smartphone


 Original message 
From: Trevor Grant 
Date: 10/16/2016 9:53 PM (GMT-08:00)
To: apache/mahout 
Cc: Andrew Palumbo , Author 
Subject: Re: [apache/mahout] MAHOUT-1885 [WIP][FOR COMMENT]: Inital 
implementation of VCL bindings for mahout math. (#261)


@rawkintrevo commented on this pull request.



In 
spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala:

> @@ -162,7 +163,7 @@ class MahoutSparkILoop extends SparkILoop {
  _ __ ___   __ _| |__   ___  _   _| |_
 | '_ ` _ \ / _` | '_ \ / _ \| | | | __|
 | | | | | | (_| | | | | (_) | |_| | |_
-|_| |_| |_|\__,_|_| |_|\___/ \__,_|\__|  version 0.12.2
+|_| |_| |_|\__,_|_| |_|\___/ \__,_|\__|  version 0.13.0



I agree to keep. The Zeppelin integration has a few of its own little 
issues, esp when it comes to serialization, and some other stuff. The 
mahout-shell will always be way more 'bullet proof'. I also like to sanity 
check myself, to figure out is an issues 1) Zeppelin's Fault, 2) Mahout's Fault 
3) My Fault, by running code in the mahout-spark shell (after a zeppelin 
failure).

-
You are receiving this because you authored the thread.
Reply to this email directly, view it on 
GitHub, or mute the 
thread.



> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Trying out PR 261

2016-10-17 Thread Andrew Musselman

Using the viennacl-omp profile I'm getting an out of memory error even with
Xmx set in MAHOUT_OPTS to 1g:

- mmul microbenchmark
  + Mahout multiplication time: 8920 ms.
  + ViennaCL/cpu/OpenMP multiplication time: 616 ms.
- trans
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[WARN] Unable to create class GPUMMul: attempting OpenMP version
[INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
[INFO] Unable to create class OMPMMul: falling back to java version
*** RUN ABORTED ***
  java.lang.OutOfMemoryError: Java heap space
  at
it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.rehash(Int2DoubleOpenHashMap.java:1059)
  at
it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.insert(Int2DoubleOpenHashMap.java:295)
  at
it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.put(Int2DoubleOpenHashMap.java:301)
  at
org.apache.mahout.math.RandomAccessSparseVector.setQuick(RandomAccessSparseVector.java:130)
  at
org.apache.mahout.math.SparseRowMatrix.setQuick(SparseRowMatrix.java:105)
  at org.apache.mahout.math.AbstractMatrix.assign(AbstractMatrix.java:256)
  at
org.apache.mahout.math.scalabindings.MatrixOps.$colon$eq(MatrixOps.scala:192)
  at
org.apache.mahout.math.scalabindings.MatrixOps.cloned(MatrixOps.scala:260)
  at
org.apache.mahout.math.scalabindings.MatrixOps.$minus(MatrixOps.scala:66)
  at
org.apache.mahout.viennacl.openmp.ViennaCLSuiteOMP$$anonfun$4.apply$mcV$sp(ViennaCLSuiteOMP.scala:152)
  ...
[INFO]

[INFO] Reactor Summary:
[INFO]
[INFO] Mahout Build Tools . SUCCESS [
 1.131 s]
[INFO] Apache Mahout .. SUCCESS [
 0.023 s]
[INFO] Mahout Math  SUCCESS [
31.439 s]
[INFO] Mahout HDFS  SUCCESS [
 2.151 s]
[INFO] Mahout Math Scala bindings . SUCCESS [02:30
min]
[INFO] Mahout Spark bindings .. SUCCESS [
43.884 s]
[INFO] Mahout Flink bindings .. SUCCESS [04:13
min]
[INFO] Mahout Spark bindings shell  SUCCESS [
 4.071 s]
[INFO] Mahout Release Package . SUCCESS [
 1.311 s]
[INFO] Mahout H2O backend . SUCCESS [02:50
min]
[INFO] Mahout Native VienniaCL OpenMP Bindings  FAILURE [
36.106 s]

On Sun, Oct 16, 2016 at 9:43 PM, Trevor Grant 
wrote:

> NOTE:
>
> The last solution creates a cyclical dependency-
>
> The way I made it work was happen-stance.
>
> 1)
> mvn clean install -Pviennacl -Phadoop2 -DskipTests
>
> 2)
> update the pom as described previously
>
> 3)
> cd math-scala
>
> 4)
> mvn test
>
> that will make it work- hacky but POC.
>
> Next how to do this 'right'.
>
> tg
>
>
>
> Trevor Grant
> Data Scientist
> https://github.com/rawkintrevo
> http://stackexchange.com/users/3002022/rawkintrevo
> http://trevorgrant.org
>
> *"Fortunate is he, who is able to know the causes of things."  -Virgil*
>
>
> On Sun, Oct 16, 2016 at 11:29 PM, Trevor Grant 
> wrote:
>
> > Maybe not the long term fix, but adding
> >
> > 
> >  org.apache.mahout
> >  mahout-native-viennacl_${scala.compat.
> > version}
> >  test
> >
> >
> > to math-scala/pom.xml seems to fix the afore mentioned errors
> >
> >
> > Trevor Grant
> > Data Scientist
> > https://github.com/rawkintrevo
> > http://stackexchange.com/users/3002022/rawkintrevo
> > http://trevorgrant.org
> >
> > *"Fortunate is he, who is able to know the causes of things."  -Virgil*
> >
> >
> > On Sun, Oct 16, 2016 at 10:23 PM, Andrew Palumbo 
> > wrote:
> >
> >> Thanks Trevor and Andrew- I think that both of your environments are
> >> fine.. (Ubuntu 16.04 was just to recommended because ViennaCL 1.7.1 is
> >> easily installed with sudo-apt-get install.  As logn as you have 1.7.1
> on
> >> the correct path, you should be fine.
> >>
> >>
> >> I'm getting these errors now also, and by viewing the stack trace, I can
> >> see that the mahout_native_vienniacl*.jars did not make it onto the
> >> classpath some how:
> >>
> >>
> >> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> >> [WARN] Unable to create class GPUMMul: attempting OpenMP version
> >> java.lang.ClassNotFoundException: org.apache.mahout.viennacl.ope
> >> ncl.GPUMMul$
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> >> at java.lang.Class.forName0(Native Method)
> >> at java.lang.Class.forName(Class.java:264)
> >> at org.apache.mahout.math.scalabindings.SolverFactory$.getOpera
> >> tor(SolverFactory.scala:50)
> >>
> >>
> >> I think that i might have merged some conflicts badly

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582587#comment-15582587
 ] 

ASF GitHub Bot commented on MAHOUT-1885:


Github user dlyubimov commented on the issue:

https://github.com/apache/mahout/pull/261
  
IMO this needs more work and committing this is premature at this point due 
to architecture considerations. Not only it misses some things, but also IMO 
has things that should not be done the way they currently are.


> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

2016-10-17 Thread Suneel Marthi (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582075#comment-15582075
 ] 

Suneel Marthi commented on MAHOUT-1885:
---

This has been fixed as of today in ViennaCL. Please update ur ViennaCL 
binaries. 

> Inital Implementation of VCL Bindings
> -
>
> Key: MAHOUT-1885
> URL: https://issues.apache.org/jira/browse/MAHOUT-1885
> Project: Mahout
>  Issue Type: Improvement
>  Components: Math
>Affects Versions: 0.12.2
>Reporter: Andrew Palumbo
>Assignee: Andrew Palumbo
> Fix For: 0.13.0
>
>
> Push a working experimental branch of VCL bindings into master.  There is 
> still a lot of work to be done.  All tests are passing, At the moment there 
> am opening this JIRA mostly to get a number for PR and to test profiles 
> against on travis. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAHOUT-1889) Mahout doesn't work with Spark 2.0

2016-10-17 Thread Suneel Marthi (JIRA)


[ 
https://issues.apache.org/jira/browse/MAHOUT-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582071#comment-15582071
 ] 

Suneel Marthi commented on MAHOUT-1889:
---

Thanks for reporting this, it makes sense to work on this if any of the Hadoop 
vendors are already packaging Spark 2x in their distros. Feel free to submit a 
patch nevertheless. 

> Mahout doesn't work with Spark 2.0
> --
>
> Key: MAHOUT-1889
> URL: https://issues.apache.org/jira/browse/MAHOUT-1889
> Project: Mahout
>  Issue Type: Bug
>Reporter: Sergey Svinarchuk
>
> In Spark 2.0 was changes path to libraries and classpath. If change classpath 
> to correct for Spark 2.0, all Spark job failed with 
> java.lang.NoSuchMethodError: , because Spark API was changed.
> Example for spark-shell:
> {code}
> ./bin/mahout spark-shell
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.apache.spark.repl.SparkILoop.setPrompt(Ljava/lang/String;)V
>   at 
> org.apache.mahout.sparkbindings.shell.MahoutSparkILoop.(MahoutSparkILoop.scala:58)
>   at org.apache.mahout.sparkbindings.shell.Main$.main(Main.scala:32)
>   at org.apache.mahout.sparkbindings.shell.Main.main(Main.scala)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAHOUT-1889) Mahout doesn't work with Spark 2.0

2016-10-17 Thread Sergey Svinarchuk (JIRA)

Sergey Svinarchuk created MAHOUT-1889:
-

 Summary: Mahout doesn't work with Spark 2.0
 Key: MAHOUT-1889
 URL: https://issues.apache.org/jira/browse/MAHOUT-1889
 Project: Mahout
  Issue Type: Bug
Reporter: Sergey Svinarchuk


In Spark 2.0 was changes path to libraries and classpath. If change classpath 
to correct for Spark 2.0, all Spark job failed with 
java.lang.NoSuchMethodError: , because Spark API was changed.

Example for spark-shell:
{code}
./bin/mahout spark-shell

Exception in thread "main" java.lang.NoSuchMethodError: 
org.apache.spark.repl.SparkILoop.setPrompt(Ljava/lang/String;)V
at 
org.apache.mahout.sparkbindings.shell.MahoutSparkILoop.(MahoutSparkILoop.scala:58)
at org.apache.mahout.sparkbindings.shell.Main$.main(Main.scala:32)
at org.apache.mahout.sparkbindings.shell.Main.main(Main.scala)

{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Trying out PR 261

Re: Trying out PR 261

RE: Trying out PR 261

Re: Trying out PR 261

RE: Trying out PR 261

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

Re: Trying out PR 261

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

[jira] [Commented] (MAHOUT-1885) Inital Implementation of VCL Bindings

[jira] [Commented] (MAHOUT-1889) Mahout doesn't work with Spark 2.0

[jira] [Created] (MAHOUT-1889) Mahout doesn't work with Spark 2.0

13 matches

Site Navigation

Mail list logo

Footer information