Not direct answer to your question. But It might be useful for you to check Spring XD Spark integration.
https://github.com/spring-projects/spring-xd-samples/tree/master/spark-streaming-wordcount-java-processor On Mon, Nov 16, 2015 at 6:14 AM, Muthu Jayakumar <bablo...@gmail.com> wrote: > I have only written Akka code in Scala only. Here is the akka > documentation that would help you to get started... > http://doc.akka.io/docs/akka/2.4.0/intro/getting-started.html > > >JavaSparkContext(conf) > The idea is to create a SparkContext and pass it as a props (constructor > in java sense) to an akka actor so that you can send interactive spark jobs > to the actor system. The only caveat is that, you'll have to run this spark > application in client mode. > > >sc.parallelize(list).foreach > >// here we will have db transaction as well. > The way I had done DB Transaction is to run a synchronous (awaitable call > from Akka sense) to perform db operation atomic to the data being processed > using slick (http://slick.typesafe.com/doc/3.1.0/gettingstarted.html). > In your case the following two links could shed some light... > - > http://stackoverflow.com/questions/24896233/how-to-save-apache-spark-schema-output-in-mysql-database > - > https://databricks.gitbooks.io/databricks-spark-reference-applications/content/logs_analyzer/chapter3/save_an_rdd_to_a_database.html > > On a side note, I noticed that you provide a custom serializer. In my > case, I have used case classes (a construct from Scala) that can use the > default serializer provided by Spark. > > Hope this helps. > > Thanks, > Muthu > > > On Sat, Nov 14, 2015 at 10:18 PM, Netai Biswas <mail2efo...@gmail.com> > wrote: > >> Hi, >> >> Thanks for your response. I will give a try with akka also, if you have >> any sample code or useful link please do share with me. Anyway I am sharing >> one sample code of mine. >> >> Sample Code: >> >> @Autowiredprivate SpringBean springBean; >> public void test() throws Exception { >> SparkConf conf = new SparkConf().setAppName("APP").setMaster(masterURL); >> conf.set("spark.serializer", >> "de.paraplu.springspark.serialization.SpringAwareSerializer"); >> sc = new JavaSparkContext(conf); >> >> sc.parallelize(list).foreach(new VoidFunction<String>() { >> private static final long serialVersionUID = 1L; >> >> @Override >> public void call(String t) throws Exception { >> springBean.someAPI(t); // here we will have db transaction as >> well. >> } >> });} >> >> Thanks, >> Netai >> >> On Sat, Nov 14, 2015 at 10:40 PM, Muthu Jayakumar <bablo...@gmail.com> >> wrote: >> >>> You could try to use akka actor system with apache spark, if you are >>> intending to use it in online / interactive job execution scenario. >>> >>> On Sat, Nov 14, 2015, 08:19 Sabarish Sasidharan < >>> sabarish.sasidha...@manthan.com> wrote: >>> >>>> You are probably trying to access the spring context from the executors >>>> after initializing it at the driver. And running into serialization issues. >>>> >>>> You could instead use mapPartitions() and initialize the spring context >>>> from within that. >>>> >>>> That said I don't think that will solve all of your issues because you >>>> won't be able to use the other rich transformations in Spark. >>>> >>>> I am afraid these two don't gel that well, unless and otherwise all >>>> your context lookups for beans happen in the driver. >>>> >>>> Regards >>>> Sab >>>> On 13-Nov-2015 4:17 pm, "Netai Biswas" <mail2efo...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I am facing issue while integrating spark with spring. >>>>> >>>>> I am getting "java.lang.IllegalStateException: Cannot deserialize >>>>> BeanFactory with id" errors for all beans. I have tried few solutions >>>>> available in web. Please help me out to solve this issue. >>>>> >>>>> Few details: >>>>> >>>>> Java : 8 >>>>> Spark : 1.5.1 >>>>> Spring : 3.2.9.RELEASE >>>>> >>>>> Please let me know if you need more information or any sample code. >>>>> >>>>> Thanks, >>>>> Netai >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-and-Spring-Integrations-tp25375.html >>>>> Sent from the Apache Spark User List mailing list archive at >>>>> Nabble.com. >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>> >>>>> >> >
