I think you could use `repartition` to make sure there would be no empty partitions.
You could also try `coalesce` to combine partitions , but it can't make sure there are no more empty partitions. Best Regards, Yi Tian [email protected] On Oct 18, 2014, at 20:30, [email protected] wrote: > Hi, > > I am developing program using Spark where I am using filter such as: > > cleanedData = distData.map(json_extractor.extract_json).filter(lambda x: x != > None and x != '') > cleanedData.saveAsTextFile(sys.argv[3]) > > > It happens to me that there is saved lot of empty files (probably from those > partitions that should have been filtered out). Is there some way, how to > prevent Spark from saving these empty files? > > Thank you in advance for any help. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
