Hi Nikita, We are actively working on this: https://github.com/cerndb/dist-keras This will allow you to run Keras on Spark (with distributed optimization algorithms) through pyspark. I recommend you to check the examples https://github.com/cerndb/dist-keras/tree/master/examples. However, you need to be aware that distributed optimization is a research topic, and has several approaches and caveats you need to be aware of. I wrote a blog post on this if you like to have some additional information on this topic https://db-blog.web.cern.ch/blog/joeri-hermans/2017-01-distributed-deep-learning-apache-spark-and-keras
However, if you don't want to use a distributed optimization algorithm, we also support a "sequential trainer" which allows you to train a model on Spark dataframes. Kind regards, Joeri ________________________________________. From: Nick Pentreath [nick.pentre...@gmail.com] Sent: 23 February 2017 13:39 To: dev@spark.apache.org Subject: Re: Implementation of RNN/LSTM in Spark The short answer is there is none and highly unlikely to be inside of Spark MLlib any time in the near future. The best bets are to look at other DL libraries - for JVM there is Deeplearning4J and BigDL (there are others but these seem to be the most comprehensive I have come across) - that run on Spark. Also there are various flavours of TensorFlow / Caffe on Spark. And of course the libs such as Torch, Keras, Tensorflow, MXNet, Caffe etc. Some of them have Java or Scala APIs and some form of Spark integration out there in the community (in varying states of development). Integrations with Spark are a bit patchy currently but include the "XOnSpark" flavours mentioned above and TensorFrames (again, there may be others). On Thu, 23 Feb 2017 at 14:23 n1kt0 <nikita.balysc...@googlemail.com<mailto:nikita.balysc...@googlemail.com>> wrote: Hi, can anyone tell me what the current status about RNNs in Spark is? -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Implementation-of-RNN-LSTM-in-Spark-tp14866p21060.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org> --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
