Hi Jacek, Thanks for the confirmation! Let's change the wording first, and open a JIRA ticket for the relative path support.
Wenchen On Tue, Apr 29, 2025 at 2:41 AM Jacek Laskowski <ja...@japila.pl> wrote: > Hi Wenchen, > > Looks like it didn't work in 3.5 either. > > ❯ ./bin/spark-shell --version > 25/04/28 20:37:48 WARN Utils: Your hostname, Jaceks-Mac-mini.local > resolves to a loopback address: 127.0.0.1; using 192.168.68.100 instead (on > interface en1) > 25/04/28 20:37:48 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to > another address > Welcome to > ____ __ > / __/__ ___ _____/ /__ > _\ \/ _ \/ _ `/ __/ '_/ > /___/ .__/\_,_/_/ /_/\_\ version 3.5.5 > /_/ > > Using Scala version 2.12.18, OpenJDK 64-Bit Server VM, 17.0.15 > Branch HEAD > Compiled by user ubuntu on 2025-02-23T20:30:46Z > Revision 7c29c664cdc9321205a98a14858aaf8daaa19db2 > Url https://github.com/apache/spark > Type --help for more information. > > ❯ ./bin/spark-shell --conf spark.jars.ivy=~/.ivy2 > 25/04/28 20:37:54 WARN Utils: Your hostname, Jaceks-Mac-mini.local > resolves to a loopback address: 127.0.0.1; using 192.168.68.100 instead (on > interface en1) > 25/04/28 20:37:54 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to > another address > Exception in thread "main" java.lang.IllegalArgumentException: basedir > must be absolute: ~/.ivy2/local > at org.apache.ivy.util.Checks.checkAbsolute(Checks.java:48) > at > org.apache.ivy.plugins.repository.file.FileRepository.setBaseDir(FileRepository.java:137) > at > org.apache.ivy.plugins.repository.file.FileRepository.<init>(FileRepository.java:44) > at > org.apache.spark.deploy.SparkSubmitUtils$.createRepoResolvers(SparkSubmit.scala:1274) > at > org.apache.spark.deploy.SparkSubmitUtils$.buildIvySettings(SparkSubmit.scala:1381) > at > org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:182) > at > org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:339) > at org.apache.spark.deploy.SparkSubmit.org > $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:969) > at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > We should change the wording in the migration guide and fix it (at least > report it as an issue). I can do both if allowed :wink: > > Pozdrawiam, > Jacek Laskowski > ---- > "The Internals Of" Online Books <https://books.japila.pl/> > Follow me on Bluesky <https://bsky.app/profile/books.japila.pl> > > <https://twitter.com/jaceklaskowski> > > > On Mon, Apr 28, 2025 at 5:04 AM Wenchen Fan <cloud0...@gmail.com> wrote: > >> Hi Jacek, >> >> Thanks for reporting the issue! Did you hit the same problem when you set >> the `spark.jars.ivy` config with Spark 3.5? If this config never worked >> with a relative path, we should change the wording in the migration guide. >> >> Thanks, >> Wenchen >> >> On Sun, Apr 27, 2025 at 10:27 PM Jacek Laskowski <ja...@japila.pl> wrote: >> >>> Hi, >>> >>> I found in docs/core-migration-guide.md: >>> >>> - Since Spark 4.0, Spark uses `~/.ivy2.5.2` as Ivy user directory by >>> default to isolate the existing systems from Apache Ivy's incompatibility. >>> To restore the legacy behavior, you can set `spark.jars.ivy` to `~/.ivy2`. >>> >>> With that, I used spark.jars.ivy to run spark-shell with ~/.ivy2 >>> directory for local dependencies. >>> >>> bin/spark-sql --conf spark.jars.ivy=~/.ivy2 >>> >>> This ended up with the following issue: >>> >>> Exception in thread "main" java.lang.IllegalArgumentException: basedir >>> must be absolute: ~/.ivy2/local >>> at org.apache.ivy.util.Checks.checkAbsolute(Checks.java:48) >>> at >>> org.apache.ivy.plugins.repository.file.FileRepository.setBaseDir(FileRepository.java:137) >>> at >>> org.apache.ivy.plugins.repository.file.FileRepository.<init>(FileRepository.java:44) >>> at >>> org.apache.spark.util.MavenUtils$.createRepoResolvers(MavenUtils.scala:159) >>> at >>> org.apache.spark.util.MavenUtils$.buildIvySettings(MavenUtils.scala:287) >>> at >>> org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:118) >>> at >>> org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:341) >>> at org.apache.spark.deploy.SparkSubmit.org >>> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:961) >>> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:204) >>> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:227) >>> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:96) >>> at >>> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1132) >>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1141) >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> >>> A workaround is to use an absolute path. >>> >>> Is this a known issue? Should I report it against rc4? Please guide. >>> Thanks! >>> >>> Pozdrawiam, >>> Jacek Laskowski >>> ---- >>> "The Internals Of" Online Books <https://books.japila.pl/> >>> Follow me on Bluesky <https://bsky.app/profile/books.japila.pl> >>> >>> <https://twitter.com/jaceklaskowski> >>> >>
