This is correct behavior. If you need to call another method simply append
another map, flatmap or whatever you need.
Depending on your use case you may use also reduce and reduce by key.
However you never (!) should use a global variable as in your snippet. This can
to work because you work in a distributed setting.
Probably the code will fail on a cluster or at random.
> On 19. Dec 2017, at 07:16, Sunitha Chennareddy <chennareddysuni...@gmail.com>
> wrote:
>
> Hi Deepak,
>
> I am able to map row to person class, issue is I want to to call another
> method.
> I tried converting to list and its not working with out using collect.
>
> Regards
> Sunitha
>> On Tuesday, December 19, 2017, Deepak Sharma <deepakmc...@gmail.com> wrote:
>> I am not sure about java but in scala it would be something like df.rdd.map{
>> x => MyClass(x.getString(0),.....)}
>>
>> HTH
>>
>> --Deepak
>>
>> On Dec 19, 2017 09:25, "Sunitha Chennareddy" <chennareddysunitha@.com> wrote:
>> Hi All,
>>
>> I am new to Spark, I want to convert DataFrame to List<JavaClass> with out
>> using collect().
>>
>> Main requirement is I need to iterate through the rows of dataframe and call
>> another function by passing column value of each row (person.getId())
>>
>> Here is the snippet I have tried, Kindly help me to resolve the issue,
>> personLst is returning 0:
>>
>> List<Person> personLst= new ArrayList<Person>();
>> JavaRDD<Person> personRDD = person_dataframe.toJavaRDD().map(new
>> Function<Row, Person>() {
>>
>> public Person call(Row row) throws Exception{
>> Person person = new Person();
>>
>> person.setId(row.getDecimal(0).longValue());
>> person.setName(row.getString(1));
>>
>> personLst.add(person);
>> // here I tried to call another
>> function but control never passed
>> return person;
>> }
>> });
>>
>> logger.info("personLst size =="+personLst.size());
>> logger.info("personRDD count ==="+personRDD.count());
>>
>> //output is
>> personLst size == 0
>> personRDD count === 3
>>
>>
