Hi Disha, Data is stacked into matrices to perform matrix-matrix multiplication (instead of matrix-vector) that is handled by native BLAS and one can get a speed-up. You can refer here for benchmarks https://github.com/fommil/netlib-java
With regards to your second question, data parallelism is handled by Spark RDD, i.e. each worker processes a subset of data partitions, and master serves the role of parameter server. Best regards, Alexander From: Disha Shrivastava [mailto:dishu....@gmail.com] Sent: Wednesday, December 30, 2015 4:03 AM To: Ulanov, Alexander Cc: dev@spark.apache.org Subject: Re: Data and Model Parallelism in MLPC Hi, I went through the code for implementation of MLPC and couldn't understand why stacking/unstacking of the input data has been done. The description says " Block size for stacking input data in matrices to speed up the computation. Data is stacked within partitions. If block size is more than remaining data in a partition then it is adjusted to the size of this data. Recommended size is between 10 and 1000. Default: 128". I am not pretty sure what this means and how does this attain speed in computation? Also, I couldn't find exactly how data parallelism as depicted in http://static.googleusercontent.com/media/research.google.com/hi//archive/large_deep_networks_nips2012.pdf<http://static.googleusercontent.com/media/research.google.com/hi/archive/large_deep_networks_nips2012.pdf> is incorporated in the existing code. There seems to be no notion of parameter server and optimization routine is also normal LBFGS not Sandblaster LBFGS. The only parallelism seems to be coming from the way input data is read and stored. Please correct me if I am wrong and clarify my doubt. Thanks and Regards, Disha On Tue, Dec 29, 2015 at 5:40 PM, Disha Shrivastava <dishu....@gmail.com<mailto:dishu....@gmail.com>> wrote: Hi Alexander, Thanks a lot for your response.Yes, I am considering the use case when the weight matrix is too large to fit into the main memory of a single machine. Can you tell me ways of dividing the weight matrix? According to my investigations so far, we can do this by two ways: 1. By parallelizing the weight matrix RDD using sc.parallelize and then using suitable map functions in the forward and backward pass. 2. By using RowMatrix / BlockMatrix to represent the weight matrix and do calculations on it. Which of these methods will be efficient to use ? Also, I came across an implementation using Akka where layer-by-layer partitioning of the network has been done (http://alexminnaar.com/implementing-the-distbelief-deep-neural-network-training-framework-with-akka.html) which I believe is model parallelism in the true sense. Please suggest any other ways/implementation that can help. I would love to hear your remarks on the above. Thanks and Regards, Disha On Wed, Dec 9, 2015 at 1:29 AM, Ulanov, Alexander <alexander.ula...@hpe.com<mailto:alexander.ula...@hpe.com>> wrote: Hi Disha, Which use case do you have in mind that would require model parallelism? It should have large number of weights, so it could not fit into the memory of a single machine. For example, multilayer perceptron topologies, that are used for speech recognition, have up to 100M of weights. Present hardware is capable of accommodating this in the main memory. That might be a problem for GPUs, but this is a different topic. The straightforward way of model parallelism for fully connected neural networks is to distribute horizontal (or vertical) blocks of weight matrices across several nodes. That means that the input data has to be reproduced on all these nodes. The forward and the backward passes will require re-assembling the outputs and the errors on each of the nodes after each layer, because each of the node can produce only partial results since it holds a part of weights. According to my estimations, this is inefficient due to large intermediate traffic between the nodes and should be used only if the model does not fit in memory of a single machine. Another way of model parallelism would be to represent the network as the graph and use GraphX to write forward and back propagation. However, this option does not seem very practical to me. Best regards, Alexander From: Disha Shrivastava [mailto:dishu....@gmail.com<mailto:dishu....@gmail.com>] Sent: Tuesday, December 08, 2015 11:19 AM To: Ulanov, Alexander Cc: dev@spark.apache.org<mailto:dev@spark.apache.org> Subject: Re: Data and Model Parallelism in MLPC Hi Alexander, Thanks for your response. Can you suggest ways to incorporate Model Parallelism in MPLC? I am trying to do the same in Spark. I got hold of your post http://apache-spark-developers-list.1001551.n3.nabble.com/Model-parallelism-with-RDD-td13141.html where you have divided the weight matrix into different worker machines. I have two basic questions in this regard: 1. How to actually visualize/analyze and control how nodes of the neural network/ weights are divided across different workers? 2. Is there any alternate way to achieve model parallelism for MPLC in Spark? I believe we need to have some kind of synchronization and control for the updation of weights shared across different workers during backpropagation. Looking forward for your views on this. Thanks and Regards, Disha On Wed, Dec 9, 2015 at 12:36 AM, Ulanov, Alexander <alexander.ula...@hpe.com<mailto:alexander.ula...@hpe.com>> wrote: Hi Disha, Multilayer perceptron classifier in Spark implements data parallelism. Best regards, Alexander From: Disha Shrivastava [mailto:dishu....@gmail.com<mailto:dishu....@gmail.com>] Sent: Tuesday, December 08, 2015 12:43 AM To: dev@spark.apache.org<mailto:dev@spark.apache.org>; Ulanov, Alexander Subject: Data and Model Parallelism in MLPC Hi, I would like to know if the implementation of MLPC in the latest released version of Spark ( 1.5.2 ) implements model parallelism and data parallelism as done in the DistBelief model implemented by Google http://static.googleusercontent.com/media/research.google.com/hi//archive/large_deep_networks_nips2012.pdf<http://static.googleusercontent.com/media/research.google.com/hi/archive/large_deep_networks_nips2012.pdf> Thanks And Regards, Disha
