Re: [mllib] Add multiplying large scale matrices

2014-09-08 Thread Xiangrui Meng
Sorry for my late reply! I'm also very interested in the
implementation of distributed matrix multiplication. As Shivaram
mentioned, the communication is the concern here. But maybe we can
start with a reasonable implementation and then iterate on its
performance. It would be great if eventually we can implement an
algorithm close to the 2.5D algorithm
(http://www.netlib.org/lapack/lawnspdf/lawn248.pdf).

I created two JIRAs for this topic:

1. Distributed block matrix: https://issues.apache.org/jira/browse/SPARK-3434
2. Distributed matrix multiplication:
https://issues.apache.org/jira/browse/SPARK-3435

We can move our discussion there.

Rong, I'm really happy to see the Saury project. It would be great if
you can share your design and experience (maybe on the JIRA page so it
is easier to track). I will read the reports on CSDN and ping you if I
ran into problems. Thanks!

Best,
Xiangrui

On Sat, Sep 6, 2014 at 1:28 AM, Yu Ishikawa
yuu.ishikawa+sp...@gmail.com wrote:
 Hi Rong,

 Great job! Thank you for let me know your work.
 I will read the source code of saury later.

 Although AMPLab is working to implement them, would you like to merge it
 into Spark?

 Best,

 -- Yu Ishikawa




 --
 View this message in context: 
 http://apache-spark-developers-list.1001551.n3.nabble.com/mllib-Add-multiplying-large-scale-matrices-tp8291p8310.html
 Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
 For additional commands, e-mail: dev-h...@spark.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org



Re: [mllib] Add multiplying large scale matrices

2014-09-08 Thread Yu Ishikawa
Hi Xiangrui Meng,

Thank you for your comment and creating tickets.

The ticket which I created would be moved to your tickets.
I will close my ticket, and then will link it to yours later.

Best,
Yu Ishikawa



--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/mllib-Add-multiplying-large-scale-matrices-tp8291p8333.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org



Re: amplab jenkins is down

2014-09-08 Thread Josh Rosen
Yeah, I think https://github.com/apache/spark/pull/2315 should have fixed
the Mima issue.  We're still seeing some intermittent failures due to
DriverSuite and SparkSubmitSuite tests failing, so I'd appreciate any help
in diagnosing that issue.

On Sun, Sep 7, 2014 at 10:08 PM, Prashant Sharma scrapco...@gmail.com
wrote:

 Looks like this is already taken care of ?

 Prashant Sharma



 On Mon, Sep 8, 2014 at 4:37 AM, Josh Rosen rosenvi...@gmail.com wrote:

 Does anyone know why some of the MiMa tests have started failing?

 See
 https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19948/consoleFull
  for
 an example.

 On September 6, 2014 at 12:48:27 PM, Josh Rosen (rosenvi...@gmail.com)
 wrote:

 It looks like Jenkins is up and running, but there seems to be a delay in
 responding to requests to re-test patches.  It seems like Jenkins is
 promptly testing new PRs, or new commits as they’re added to existing PRs,
 but taking a very long time to respond to requests to re-test PRs.

 I’m going to continue monitoring this today.  I’m considering creating my
 own fork of the Jenkins pull request builder plugin so that we can add
 extra logging in order to diagnose what’s causing this lag.

 - Josh
 On September 5, 2014 at 5:49:32 PM, Josh Rosen (rosenvi...@gmail.com)
 wrote:

 We have successfully purged Jenkins’ build queue.  If you want a PR to be
 re-tested, please ask Jenkins again.

 On September 5, 2014 at 5:36:30 PM, shane knapp (skn...@berkeley.edu)
 wrote:

 yeah, it was a problem w/the PRB's OAuth key. josh rosen added a new key,
 and magique!

 we're about to clear the queue of all builds as most aren't wanted/needed.


 On Fri, Sep 5, 2014 at 5:33 PM, Nicholas Chammas 
 nicholas.cham...@gmail.com
  wrote:

  Looks like Jenkins is back!
 
  lol The poor guy has like a million builds
  
 https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/
 
  to catch up on.
 
 
  On Fri, Sep 5, 2014 at 4:15 PM, Nicholas Chammas 
  nicholas.cham...@gmail.com wrote:
 
  How's it going?
 
  It looks like during the last build
  
 https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/lastBuild/console
 
  from about 30 min ago Jenkins was still having trouble fetching from
  GitHub. It also looks like not all requests for testing are triggering
  builds.
 
 
  On Fri, Sep 5, 2014 at 1:23 PM, shane knapp skn...@berkeley.edu
 wrote:
 
  it's looking like everything except the pull request builders are
  working. i'm going to be working on getting this resolved today.
 
 
  On Fri, Sep 5, 2014 at 8:18 AM, Nicholas Chammas 
  nicholas.cham...@gmail.com wrote:
 
  Hmm, looks like at least some builds
  
 https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/19804/consoleFull
 
  are working now, though this last one was from ~5 hours ago.
 
 
  On Fri, Sep 5, 2014 at 1:02 AM, shane knapp skn...@berkeley.edu
  wrote:
 
  yep. that's exactly the behavior i saw earlier, and will be figuring
  out first thing tomorrow morning. i bet it's an environment issues
 on the
  slaves.
 
 
  On Thu, Sep 4, 2014 at 7:10 PM, Nicholas Chammas 
  nicholas.cham...@gmail.com wrote:
 
  Looks like during the last build
  
 https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/19797/console
 
  Jenkins was unable to execute a git fetch?
 
 
  On Thu, Sep 4, 2014 at 7:58 PM, shane knapp skn...@berkeley.edu
  wrote:
 
  i'm going to restart jenkins and see if that fixes things.
 
 
  On Thu, Sep 4, 2014 at 4:56 PM, shane knapp skn...@berkeley.edu
  wrote:
 
  looking
 
 
  On Thu, Sep 4, 2014 at 4:21 PM, Nicholas Chammas 
  nicholas.cham...@gmail.com wrote:
 
  It appears that our main man is having trouble
  
 https://amplab.cs.berkeley.edu/jenkins/view/Pull%20Request%20Builders/job/SparkPullRequestBuilder/
 
  hearing new requests
  
 https://github.com/apache/spark/pull/2277#issuecomment-54549106.
 
  Do we need some smelling salts?
 
 
  On Thu, Sep 4, 2014 at 5:49 PM, shane knapp 
 skn...@berkeley.edu
  wrote:
 
  i'd ping the Jenkinsmench... the master was completely offline,
  so any new
  jobs wouldn't have reached it. any jobs that were queued when
  power was
  lost probably started up, but jobs that were running would
 fail.
 
 
  On Thu, Sep 4, 2014 at 2:45 PM, Nicholas Chammas 
  nicholas.cham...@gmail.com
   wrote:
 
   Woohoo! Thanks Shane.
  
   Do you know if queued PR builds will automatically be picked
  up? Or do we
   have to ping the Jenkinmensch manually from each PR?
  
   Nick
  
  
   On Thu, Sep 4, 2014 at 5:37 PM, shane knapp 
  skn...@berkeley.edu wrote:
  
   AND WE'RE UP!
  
   sorry that this took so long... i'll send out a more
 detailed
  explanation
   of what happened soon.
  
   now, off to back up jenkins.
  
   shane
  
  
   On Thu, Sep 4, 2014 at 1:27 PM, shane knapp 
  skn...@berkeley.edu wrote:
  
it's a