thierry accart created SPARK-30190: -------------------------------------- Summary: HistoryServerDiskManager will fail on appStoreDir in s3 Key: SPARK-30190 URL: https://issues.apache.org/jira/browse/SPARK-30190 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 2.4.4 Reporter: thierry accart
Hi While setting spark.eventLog.dir to s3a://... I realized that it *requires destination directory to preexists for S3* This is explained I think in HistoryServerDiskManager's appStoreDir: it tries check if directory exists or can be created {{if (!appStoreDir.isDirectory() && !appStoreDir.mkdir()) \{throw new IllegalArgumentException(s"Failed to create app directory ($appStoreDir).")}}} But in S3, a directory does not exists and cannot be created: directories don't exists by themselves, they are only materialized due to existence of objects. Before proposing a patch, I wanted to know what are the prefered options : should we have a spark option to skip the appStoreDir test, or skip it only when a particular scheme is set, have a custom implementation of HistoryServerDiskManager ...? _Note for people facing the {{IllegalArgumentException:}} {{Failed to create app directory}} *you just have to put an empty file in bucket destination 'path'*._ -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org