Forgot to mention: Spark 3.5.2 is used
On 2024/12/03 15:05:18 Дубинкин Егор wrote: > Hello Community, > > I need to delete old src data created by Spark Structured Streaming. > Just deleting relevant folder throws an exception while reading batch > dataframe from file-system: > > java.io.FileNotFoundException: File > file:/data/avro/year=2020/month=3/day=13/hour=12/part-00000-0cc84e65-3f49-4686-85e3-1ecf48952794.c000.avro > does not exist > > > Issue is actualy the same that described here: > https://stackoverflow.com/questions/60773445/how-to-delete-old-data-that-was-created-by-spark-structured-streaming?newreg=5cc791c48358491c88d9b2dae1e436d9 > > Didn't find a way to delete it via Spark API. > Are there any solutions to do it via API instead of editing metadata manually? > > Your help would be appreciated. >
