skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-491006649 Should spark manage s3 folders accounts etc? This is optimized for the app dev cycle where I do a change and my updated jar is uploaded and is then overwritten. Im not also not comfortable with the attitude I deal with but I guess this how things has always been. Anyway im not convinced with the argument. Btw Im not sure Spark deletes all files it generates like checkpoints because they can bw reused and only the user knows when. Still uploaded files are not generated by Spark they come from the user.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
