Hi Jorn,
In my case I have to call common interface function
by passing the values of each rdd. So I have tried iterating , but I was
not able to trigger common function from call method as commented in the
snippet code in my earlier mail.
Request you please share your views.
Regards
Sunitha
On Tuesday, December 19, 2017, Jörn Franke <jornfra...@gmail.com> wrote:
> This is correct behavior. If you need to call another method simply append
> another map, flatmap or whatever you need.
>
> Depending on your use case you may use also reduce and reduce by key.
> However you never (!) should use a global variable as in your snippet.
> This can to work because you work in a distributed setting.
> Probably the code will fail on a cluster or at random.
>
> On 19. Dec 2017, at 07:16, Sunitha Chennareddy <
> chennareddysuni...@gmail.com> wrote:
>
> Hi Deepak,
>
> I am able to map row to person class, issue is I want to to call another
> method.
> I tried converting to list and its not working with out using collect.
>
> Regards
> Sunitha
> On Tuesday, December 19, 2017, Deepak Sharma <deepakmc...@gmail.com>
> wrote:
>
>> I am not sure about java but in scala it would be something like
>> df.rdd.map{ x => MyClass(x.getString(0),.....)}
>>
>> HTH
>>
>> --Deepak
>>
>> On Dec 19, 2017 09:25, "Sunitha Chennareddy" <chennareddysunitha@.com
>> <chennareddysuni...@gmail.com>> wrote:
>>
>> Hi All,
>>
>> I am new to Spark, I want to convert DataFrame to List<JavaClass> with
>> out using collect().
>>
>> Main requirement is I need to iterate through the rows of dataframe and
>> call another function by passing column value of each row (person.getId())
>>
>> Here is the snippet I have tried, Kindly help me to resolve the issue,
>> personLst is returning 0:
>>
>> List<Person> personLst= new ArrayList<Person>();
>> JavaRDD<Person> personRDD = person_dataframe.toJavaRDD().map(new
>> Function<Row, Person>() {
>> public Person call(Row row) throws Exception{
>> Person person = new Person();
>> person.setId(row.getDecimal(0).longValue());
>> person.setName(row.getString(1));
>>
>> personLst.add(person);
>> // here I tried to call another function but control never passed
>> return person;
>> }
>> });
>> logger.info("personLst size =="+personLst.size());
>> logger.info("personRDD count ==="+personRDD.count());
>>
>> //output is
>> personLst size == 0
>> personRDD count === 3
>>
>>
>>
