Thanks a lot.That means Spark does not support the nested RDD? if I pass the javaSparkContext that also wont work. I mean passing SparkContext not possible since its not serializable
i have a requirement where I will get JavaRDD<VendorRecord> matchRdd and I need to return the postential matches for this record from Hbase. so for each field of VendorRecord I have to do following 1. query Hbase to get the list of potential record in RDD 2. run logistic regression on RDD return from steps 1 and each element of the passed matchRdd. On 7 April 2015 at 03:33, Dean Wampler <deanwamp...@gmail.com> wrote: > The "log" instance won't be serializable, because it will have a file > handle to write to. Try defining another static method outside > matchAndMerge that encapsulates the call to log.error. CompanyMatcherHelper > might not be serializable either, but you didn't provide it. If it holds a > database connection, same problem. > > You can't suppress the warning because it's actually an error. The > VoidFunction can't be serialized to send it over the cluster's network. > > dean > > Dean Wampler, Ph.D. > Author: Programming Scala, 2nd Edition > <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) > Typesafe <http://typesafe.com> > @deanwampler <http://twitter.com/deanwampler> > http://polyglotprogramming.com > > On Mon, Apr 6, 2015 at 4:30 PM, Jeetendra Gangele <gangele...@gmail.com> > wrote: > >> In this code in foreach I am getting task not serialized exception >> >> >> @SuppressWarnings("serial") >> public static void matchAndMerge(JavaRDD<VendorRecord> matchRdd, final >> JavaSparkContext jsc) throws IOException{ >> log.info("Company matcher started"); >> //final JavaSparkContext jsc = getSparkContext(); >> matchRdd.foreachAsync(new VoidFunction<VendorRecord>(){ >> @Override >> public void call(VendorRecord t) throws Exception { >> if(t !=null){ >> try{ >> CompanyMatcherHelper.UpdateMatchedRecord(jsc,t); >> } catch (Exception e) { >> log.error("ERROR while running Matcher for company " + t.getCompanyId(), >> e); >> } >> } >> } >> }); >> >> } >> > >
