Minack
Sent: Thursday, February 27, 2020 8:51 PM
To: Manjunath Shetty H ; user@spark.apache.org
Subject: Re: Convert each partition of RDD to Dataframe
Manjunath,
You can define your DataFrame in parallel in a multi-threaded driver.
Enrico
Am 27.02.20 um 15:50 schrieb Manjunath Shetty H:
Hi
What will be the performance implications ?
Regards
Manjunath
From: Enrico Minack
Sent: Thursday, February 27, 2020 7:57 PM
To:user@spark.apache.org
Subject: Re: Convert each partition of RDD to Dataframe
Hi Manjunath,
why not creating 10 DataFrames loading the different tables in the
*Sent:* Thursday, February 27, 2020 7:57 PM
*To:* user@spark.apache.org
*Subject:* Re: Convert each partition of RDD to Dataframe
Hi Manjunath,
why not creating 10 DataFrames loading the different tables in the
first place?
Enrico
Am 27.02.20 um 14:53 schrieb Manjunath Shetty H:
Hi Vinodh
will be the performance implications ?
Regards
Manjunath
From: Enrico Minack
Sent: Thursday, February 27, 2020 7:57 PM
To: user@spark.apache.org
Subject: Re: Convert each partition of RDD to Dataframe
Hi Manjunath,
why not creating 10 DataFrames loading the
user
*Subject:* Re: Convert each partition of RDD to Dataframe
Just split the single rdd into multiple individual rdds using a filter
operation and then convert each individual rdds to it's respective
dataframe..
On Thu, Feb 27, 2020, 7:29 AM Manjunath Shetty H
mailto:manjunathshe...@live.c
arles vinodh
Sent: Thursday, February 27, 2020 7:04 PM
To: manjunathshe...@live.com
Cc: user
Subject: Re: Convert each partition of RDD to Dataframe
Just split the single rdd into multiple individual rdds using a filter
operation and then convert each individual rdds to it's respective dataf
Just split the single rdd into multiple individual rdds using a filter
operation and then convert each individual rdds to it's respective
dataframe..
On Thu, Feb 27, 2020, 7:29 AM Manjunath Shetty H
wrote:
>
> Hello All,
>
> In spark i am creating the custom partitions with Custom RDD, each
> pa
