steveloughran commented on PR #39185: URL: https://github.com/apache/spark/pull/39185#issuecomment-1364000436
1. I think i should add a test which verifies the abs path temp file is always under .spark-staging *and* includes the task attempt id in it. 1. jobs created through RDDs don't get unique IDs across processes, they just get the RDD counter. so > 1 job writing to same dest may really ruin the lives of others even when updating different partitions. I can fix that here or file a separate JIRA -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
