Thanks Sean.I just realized it. Let me try that.
On Mon, Mar 22, 2021 at 12:31 PM Sean Owen wrote:
> You need to do something with the result of repartition. You haven't
> changed textDF
>
> On Mon, Mar 22, 2021, 12:15 PM KhajaAsmath Mohammed <
> mdkhajaasm...@gmail.com> wrote:
>
>> Hi,
>>
>> I
You need to do something with the result of repartition. You haven't
changed textDF
On Mon, Mar 22, 2021, 12:15 PM KhajaAsmath Mohammed
wrote:
> Hi,
>
> I have a use case where there are large files in hdfs.
>
> Size of the file is 3 GB.
>
> It is an existing code in production and I am trying t
Hi,
I have a use case where there are large files in hdfs.
Size of the file is 3 GB.
It is an existing code in production and I am trying to improve the
performance of the job.
Sample Code:
textDF=dataframe ( This is dataframe that got created from hdfs path)
logging.info("Number of partitions"
