On 7 April 2015 at 04:03, Dean Wampler <[email protected]> wrote:
> > On Mon, Apr 6, 2015 at 6:20 PM, Jeetendra Gangele <[email protected]> > wrote: > >> Thanks a lot.That means Spark does not support the nested RDD? >> if I pass the javaSparkContext that also wont work. I mean passing >> SparkContext not possible since its not serializable >> >> That's right. RDD don't nest and SparkContexts aren't serializable. > > > >> i have a requirement where I will get JavaRDD<VendorRecord> matchRdd and >> I need to return the postential matches for this record from Hbase. so for >> each field of VendorRecord I have to do following >> >> 1. query Hbase to get the list of potential record in RDD >> 2. run logistic regression on RDD return from steps 1 and each element of >> the passed matchRdd. >> >> If I understand you correctly, each VectorRecord could correspond to > 0-to-N records in HBase, which you need to fetch, true? > yes thats correct each Vendorrecord corresponds to 0 to N matches > If so, you could use the RDD flatMap method, which takes a function a > that accepts each record, then returns a sequence of 0-to-N new records of > some other type, like your HBase records. However, running an HBase query > for each VendorRecord could be expensive. If you can turn this into a range > query or something like that, it would help. I haven't used HBase much, so > I don't have good advice on optimizing this, if necessary. > > Alternatively, can you do some sort of join on the VendorRecord RDD and an > RDD of query results from HBase? > Join will give too big result RDD of query result is returning around 10000 for each record and i have 2 millions to process so it will be huge to have this. 2 m*10000 big number > > For #2, it sounds like you need flatMap to return records that combine the > input VendorRecords and fields pulled from HBase. > > Whatever you can do to make this work like "table scans" and joins will > probably be most efficient. > > dean > >> >> >> >> On 7 April 2015 at 03:33, Dean Wampler <[email protected]> wrote: >> >>> The "log" instance won't be serializable, because it will have a file >>> handle to write to. Try defining another static method outside >>> matchAndMerge that encapsulates the call to log.error. CompanyMatcherHelper >>> might not be serializable either, but you didn't provide it. If it holds a >>> database connection, same problem. >>> >>> You can't suppress the warning because it's actually an error. The >>> VoidFunction can't be serialized to send it over the cluster's network. >>> >>> dean >>> >>> Dean Wampler, Ph.D. >>> Author: Programming Scala, 2nd Edition >>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) >>> Typesafe <http://typesafe.com> >>> @deanwampler <http://twitter.com/deanwampler> >>> http://polyglotprogramming.com >>> >>> On Mon, Apr 6, 2015 at 4:30 PM, Jeetendra Gangele <[email protected]> >>> wrote: >>> >>>> In this code in foreach I am getting task not serialized exception >>>> >>>> >>>> @SuppressWarnings("serial") >>>> public static void matchAndMerge(JavaRDD<VendorRecord> matchRdd, >>>> final JavaSparkContext jsc) throws IOException{ >>>> log.info("Company matcher started"); >>>> //final JavaSparkContext jsc = getSparkContext(); >>>> matchRdd.foreachAsync(new VoidFunction<VendorRecord>(){ >>>> @Override >>>> public void call(VendorRecord t) throws Exception { >>>> if(t !=null){ >>>> try{ >>>> CompanyMatcherHelper.UpdateMatchedRecord(jsc,t); >>>> } catch (Exception e) { >>>> log.error("ERROR while running Matcher for company " + >>>> t.getCompanyId(), e); >>>> } >>>> } >>>> } >>>> }); >>>> >>>> } >>>> >>> >>> >> >> >> >> >
