Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/19851#discussion_r154027755
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
---
@@ -50,14 +52,24 @@ class HiveExternalCatalogVersionsSuite extends
SparkSubmitTestUtils {
super.afterAll()
}
- private def downloadSpark(version: String): Unit = {
- import scala.sys.process._
+ private def tryDownloadSpark(version: String, path: String): Unit = {
+ // Try mirrors a few times until one succeeds
+ for (i <- 0 until 3) {
+ val preferredMirror =
+ Seq("wget",
"https://www.apache.org/dyn/closer.lua?preferred=true";, "-q", "-O", "-").!!.trim
+ val url =
s"$preferredMirror/spark/spark-$version/spark-$version-bin-hadoop2.7.tgz"
+ logInfo(s"Downloading Spark $version from $url")
--- End diff --
it is possible to also pass `path` but from testing, it doesn't seem like
the path/path_info value is validated in any way.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
