You need to do something with the result of repartition. You haven't changed textDF
On Mon, Mar 22, 2021, 12:15 PM KhajaAsmath Mohammed <[email protected]> wrote: > Hi, > > I have a use case where there are large files in hdfs. > > Size of the file is 3 GB. > > It is an existing code in production and I am trying to improve the > performance of the job. > > Sample Code: > textDF=dataframe ( This is dataframe that got created from hdfs path) > logging.info("Number of partitions"+str(txt_df.rdd.getNumPartitions())) > --> Prints 1 > textDF.repartition(100) > logging.info("Number of partitions"+str(txt_df.rdd.getNumPartitions())) > --> Prints 1 > > Any suggestions on why this is happening? > > Next Block of the code which takes time: > rdd.filter(lambda line: len(line)!=collistlenth) > > any way to parallelize and speed up my process on this? > > Thanks, > Asmath >
