dongjoon-hyun commented on code in PR #38376:
URL: https://github.com/apache/spark/pull/38376#discussion_r1069923212
##########
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala:
##########
@@ -239,6 +239,41 @@ private[spark] trait DepsTestsSuite { k8sSuite:
KubernetesSuite =>
}
}
+ test(
+ "SPARK-40817: Check that remote files do not get discarded in spark.files",
+ k8sTestTag,
+ MinikubeTag) {
+ tryDepsTest({
+ // Create a local file
+ val localFileName = Utils.createTempFile(FILE_CONTENTS, HOST_PATH)
+
+ // Create a remote file on S3
+ val remoteFileKey = "some-path/some-remote-file.txt"
+ createS3Object(remoteFileKey, "Some Content")
+ val remoteFileFullPath = s"s3a://${BUCKET}/${remoteFileKey}"
+
+ // Put both file paths in spark.files
+ sparkAppConf.set("spark.files",
s"$HOST_PATH/$localFileName,${remoteFileFullPath}")
+ // Allows to properly read executor logs once the job is finished
+ sparkAppConf.set("spark.kubernetes.executor.deleteOnTermination",
"false")
Review Comment:
Do we need this for this automated unit test case when we have
`expectedExecutorLogOnCompletion`? I'm wondering if this is for human.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
