On 7 April 2015 at 04:03, Dean Wampler <[email protected]> wrote:

>
> On Mon, Apr 6, 2015 at 6:20 PM, Jeetendra Gangele <[email protected]>
> wrote:
>
>> Thanks a lot.That means Spark does not support the nested RDD?
>> if I pass the javaSparkContext that also wont work. I mean passing
>> SparkContext not possible since its not serializable
>>
>> That's right. RDD don't nest and SparkContexts aren't serializable.
> 
>
>
>> i have a requirement where I will get JavaRDD<VendorRecord> matchRdd and
>> I need to return the postential matches for this record from Hbase. so for
>> each field of VendorRecord I have to do following
>>
>> 1. query Hbase to get the list of potential record in RDD
>> 2. run logistic regression on RDD return from steps 1 and each element of
>> the passed matchRdd.
>>
>> If I understand you correctly, each VectorRecord could correspond to
> 0-to-N records in HBase, which you need to fetch, true?
>
 yes thats correct each Vendorrecord corresponds to 0 to N matches


> If so, you could use the RDD flatMap method, which takes a function a
> that accepts each record, then returns a sequence of 0-to-N new records of
> some other type, like your HBase records. However, running an HBase query
> for each VendorRecord could be expensive. If you can turn this into a range
> query or something like that, it would help. I haven't used HBase much, so
> I don't have good advice on optimizing this, if necessary.
>
> Alternatively, can you do some sort of join on the VendorRecord RDD and an
> RDD of query results from HBase?
>
 Join will give too big result RDD of query result is returning around
10000 for each record and i have 2 millions to process so it will be huge
to have this. 2 m*10000 big number

>
> For #2, it sounds like you need flatMap to return records that combine the
> input VendorRecords and fields pulled from HBase.
>
> Whatever you can do to make this work like "table scans" and joins will
> probably be most efficient.
>
> dean
>
>>
>>
>>
>> On 7 April 2015 at 03:33, Dean Wampler <[email protected]> wrote:
>>
>>> The "log" instance won't be serializable, because it will have a file
>>> handle to write to. Try defining another static method outside
>>> matchAndMerge that encapsulates the call to log.error. CompanyMatcherHelper
>>> might not be serializable either, but you didn't provide it. If it holds a
>>> database connection, same problem.
>>>
>>> You can't suppress the warning because it's actually an error. The
>>> VoidFunction can't be serialized to send it over the cluster's network.
>>>
>>> dean
>>>
>>> Dean Wampler, Ph.D.
>>> Author: Programming Scala, 2nd Edition
>>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>>> Typesafe <http://typesafe.com>
>>> @deanwampler <http://twitter.com/deanwampler>
>>> http://polyglotprogramming.com
>>>
>>> On Mon, Apr 6, 2015 at 4:30 PM, Jeetendra Gangele <[email protected]>
>>> wrote:
>>>
>>>> In this code in foreach I am getting task not serialized exception
>>>>
>>>>
>>>> @SuppressWarnings("serial")
>>>> public static  void  matchAndMerge(JavaRDD<VendorRecord> matchRdd,
>>>>  final JavaSparkContext jsc) throws IOException{
>>>> log.info("Company matcher started");
>>>> //final JavaSparkContext jsc = getSparkContext();
>>>>   matchRdd.foreachAsync(new VoidFunction<VendorRecord>(){
>>>> @Override
>>>> public void call(VendorRecord t) throws Exception {
>>>>  if(t !=null){
>>>> try{
>>>> CompanyMatcherHelper.UpdateMatchedRecord(jsc,t);
>>>>  } catch (Exception e) {
>>>> log.error("ERROR while running Matcher for company " +
>>>> t.getCompanyId(), e);
>>>> }
>>>> }
>>>>  }
>>>> });
>>>>
>>>>  }
>>>>
>>>
>>>
>>
>>
>>
>>
>

Reply via email to