Re: ivy unit test case filing for Spark
Are you using IvyVPN which causes this problem? If the VPN software changes the network URL silently you should avoid using them. Regards. On Wed, Dec 22, 2021 at 1:48 AM Pralabh Kumar wrote: > Hi Spark Team > > I am building a spark in VPN . But the unit test case below is failing. > This is pointing to ivy location which cannot be reached within VPN . Any > help would be appreciated > > test("SPARK-33084: Add jar support Ivy URI -- default transitive = true") > { > *sc *= new SparkContext(new > SparkConf().setAppName("test").setMaster("local-cluster[3, > 1, 1024]")) > *sc*.addJar("*ivy://org.apache.hive:hive-storage-api:2.7.0*") > assert(*sc*.listJars().exists(_.contains( > "org.apache.hive_hive-storage-api-2.7.0.jar"))) > assert(*sc*.listJars().exists(_.contains( > "commons-lang_commons-lang-2.6.jar"))) > } > > Error > > - SPARK-33084: Add jar support Ivy URI -- default transitive = true *** > FAILED *** > java.lang.RuntimeException: [unresolved dependency: > org.apache.hive#hive-storage-api;2.7.0: not found] > at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates( > SparkSubmit.scala:1447) > at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies( > DependencyUtils.scala:185) > at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies( > DependencyUtils.scala:159) > at org.apache.spark.SparkContext.addJar(SparkContext.scala:1996) > at org.apache.spark.SparkContext.addJar(SparkContext.scala:1928) > at org.apache.spark.SparkContextSuite.$anonfun$new$115(SparkContextSuite. > scala:1041) > at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) > at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) > at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) > at org.scalatest.Transformer.apply(Transformer.scala:22) > > Regards > Pralabh Kumar > > >
Re: ivy unit test case filing for Spark
You would have to make it available? This doesn't seem like a spark issue. On Tue, Dec 21, 2021, 10:48 AM Pralabh Kumar wrote: > Hi Spark Team > > I am building a spark in VPN . But the unit test case below is failing. > This is pointing to ivy location which cannot be reached within VPN . Any > help would be appreciated > > test("SPARK-33084: Add jar support Ivy URI -- default transitive = true") > { > *sc *= new SparkContext(new > SparkConf().setAppName("test").setMaster("local-cluster[3, > 1, 1024]")) > *sc*.addJar("*ivy://org.apache.hive:hive-storage-api:2.7.0*") > assert(*sc*.listJars().exists(_.contains( > "org.apache.hive_hive-storage-api-2.7.0.jar"))) > assert(*sc*.listJars().exists(_.contains( > "commons-lang_commons-lang-2.6.jar"))) > } > > Error > > - SPARK-33084: Add jar support Ivy URI -- default transitive = true *** > FAILED *** > java.lang.RuntimeException: [unresolved dependency: > org.apache.hive#hive-storage-api;2.7.0: not found] > at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates( > SparkSubmit.scala:1447) > at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies( > DependencyUtils.scala:185) > at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies( > DependencyUtils.scala:159) > at org.apache.spark.SparkContext.addJar(SparkContext.scala:1996) > at org.apache.spark.SparkContext.addJar(SparkContext.scala:1928) > at org.apache.spark.SparkContextSuite.$anonfun$new$115(SparkContextSuite. > scala:1041) > at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) > at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) > at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) > at org.scalatest.Transformer.apply(Transformer.scala:22) > > Regards > Pralabh Kumar > > >
ivy unit test case filing for Spark
Hi Spark Team I am building a spark in VPN . But the unit test case below is failing. This is pointing to ivy location which cannot be reached within VPN . Any help would be appreciated test("SPARK-33084: Add jar support Ivy URI -- default transitive = true") { *sc *= new SparkContext(new SparkConf().setAppName("test").setMaster("local-cluster[3, 1, 1024]")) *sc*.addJar("*ivy://org.apache.hive:hive-storage-api:2.7.0*") assert(*sc*.listJars().exists(_.contains( "org.apache.hive_hive-storage-api-2.7.0.jar"))) assert(*sc*.listJars().exists(_.contains( "commons-lang_commons-lang-2.6.jar"))) } Error - SPARK-33084: Add jar support Ivy URI -- default transitive = true *** FAILED *** java.lang.RuntimeException: [unresolved dependency: org.apache.hive#hive-storage-api;2.7.0: not found] at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates( SparkSubmit.scala:1447) at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies( DependencyUtils.scala:185) at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies( DependencyUtils.scala:159) at org.apache.spark.SparkContext.addJar(SparkContext.scala:1996) at org.apache.spark.SparkContext.addJar(SparkContext.scala:1928) at org.apache.spark.SparkContextSuite.$anonfun$new$115(SparkContextSuite. scala:1041) at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) Regards Pralabh Kumar