antonipp commented on code in PR #38376:
URL: https://github.com/apache/spark/pull/38376#discussion_r1004422533
##########
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala:
##########
@@ -1609,6 +1609,16 @@ class TestFileSystem extends
org.apache.hadoop.fs.LocalFileSystem {
}
override def open(path: Path): FSDataInputStream = super.open(local(path))
+
+ // No-op methods
+
+ override def copyFromLocalFile(
Review Comment:
This was needed for the unit test to work. I was looking for a way to mock
the
[KubernetesUtils.uploadFileUri()](https://github.com/apache/spark/blob/a27b459be3ca2ad2d50b9d793b939071ca2270e2/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L310-L335)
method which is used to actually upload the local files to a remote path
specified in `spark.kubernetes.file.upload.path`. `KubernetesUtils` is an
`object` which are notoriously hard to mock and there was a bit too much
refactoring required to make this method mockable.
So I decided to go with another approach: use a fake Hadoop FS so that the
operations performed in the `uploadFileUri()` method become no-ops. I found
this `TestFileSystem` class which was used in other tests for the same purpose
so I simply added the methods which were called inside `uploadFileUri()`:
`mkdirs`
[here](https://github.com/apache/spark/blob/a27b459be3ca2ad2d50b9d793b939071ca2270e2/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L320)
and `copyFromLocalFile`
[here](https://github.com/apache/spark/blob/a27b459be3ca2ad2d50b9d793b939071ca2270e2/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L347).
I then made sure that this FS implementation was used by the test by
specifying it here:
https://github.com/antonipp/spark/blob/80416f6a6ff42051cd38dfe32df2d0c7cd0741f4/resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep
Suite.scala#L358
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
