Re: Which linear algebra interface to use within Spark MLlib?

dgemm dgemv and dot come to Breeze and Spark through netlib-java Right now both in dot and dgemv Breeze does a extra memory allocate but we already found the issue and we are working on adding a common trait that will provide a sink operation (basically memory will be allocated by user)...addi

SparkSQL 1.3.0 JDBC data source issues

Hi, I am trying jdbc data source in spark sql 1.3.0 and found some issues. First, the syntax "where str_col='value'" will give error for both postgresql and mysql: psql> create table foo(id int primary key,name text,age int); bash> SPARK_CLASSPATH=postgresql-9.4-1201-jdbc41.jar spark/bin/spark-s

Which linear algebra interface to use within Spark MLlib?

Hi, Currently I am using Breeze within Spark MLlib for linear algebra. I would like to reuse previously allocated matrices for storing the result of matrices multiplication, i.e. I need to use "gemm" function C:=q*A*B+p*C, which is missing in Breeze (Breeze automatically allocates a new matrix

Re: Block Transfer Service encryption support

Still looking for feedback... I opened the ticket as a minor, thinking of changing it to a major? Anyone object. I did see that a related ticket (https://issues.apache.org/jira/browse/SPARK-6229) is marked as a major... Cheers! Jeff -- View this message in context: http://apache-spark-develop

Re: parquet support - some questions about code

Hey Gil, ParquetRelation2 is based on the external data sources API, which is a more modular and non-intrusive way to add external data sources to Spark SQL. We are planning to replace ParquetRelation with ParquetRelation2 entirely after the latter is more mature and stable. That's why you see

parquet support - some questions about code

Hi, I am trying to better understand the code for Parquet support. In particular i got lost trying to understand ParquetRelation and ParquetRelation2. Does ParquetRelation2 is the new code that should completely remove ParquetRelation? ( I think there is some remark in the code notifying this

Re: [mllib] Is there any bugs to divide a Breeze sparse vectors at Spark v1.3.0-rc3?

Sorry for the delay in replying. I moved from Tokyo to New York in order to attend Spark Summit East. I verified the snapshot and the difference. https://github.com/scalanlp/breeze/commit/f61d2f61137807651fc860404a244640e213f6d3 Thank you for your great work! Yu Ishikawa - -- Yu Ishikawa --

Re: Using Spark with a SOCKS proxy

Did you try ssh tunneling instead of SOCKS? Thanks Best Regards On Wed, Mar 18, 2015 at 5:45 AM, Kelly, Jonathan wrote: > I'm trying to figure out how I might be able to use Spark with a SOCKS > proxy. That is, my dream is to be able to write code in my IDE then run it > without much trouble

Re: Fixed worker ports in the spark worker

Yes On Wed, Mar 18, 2015 at 1:35 PM, Niranda Perera wrote: > Thanks Arush. > > this is governed by the conf/spark-defaults.conf config, is it? > > On Wed, Mar 18, 2015 at 1:30 PM, Arush Kharbanda < > ar...@sigmoidanalytics.com> wrote: > >> You can fix the ports in the configuration - >> >> http:

Re: Fixed worker ports in the spark worker

Thanks Arush. this is governed by the conf/spark-defaults.conf config, is it? On Wed, Mar 18, 2015 at 1:30 PM, Arush Kharbanda wrote: > You can fix the ports in the configuration - > > http://spark.apache.org/docs/1.2.0/configuration.html#networking > > On Wed, Mar 18, 2015 at 11:10 AM, Niranda

Re: [mllib] Is there any bugs to divide a Breeze sparse vectors at Spark v1.3.0-rc3?

sure. On Wed, Mar 18, 2015 at 12:19 AM, Debasish Das wrote: > Hi David, > > We are stress testing breeze.optimize.proximal and nnls...if you are > cutting a release now, we will need another release soon once we get the > runtime optimizations in place and merged to breeze. > > Thanks. > Deb >

Re: Fixed worker ports in the spark worker

You can fix the ports in the configuration - http://spark.apache.org/docs/1.2.0/configuration.html#networking On Wed, Mar 18, 2015 at 11:10 AM, Niranda Perera wrote: > Hi all, > > I see that spark server opens up random ports, especially in the workers. > > is there any way to fix these ports o

Re: [mllib] Is there any bugs to divide a Breeze sparse vectors at Spark v1.3.0-rc3?

Hi David, We are stress testing breeze.optimize.proximal and nnls...if you are cutting a release now, we will need another release soon once we get the runtime optimizations in place and merged to breeze. Thanks. Deb On Mar 15, 2015 9:39 PM, "David Hall" wrote: > snapshot is pushed. If you ver