> > ... but I'm not sure that running different code with --jar and --packages > is a good idea, even if the differences are only in references to shaded > classes.
If one works without shading, why can the other not work without shading? I agreed that it should work for both consistently. That's what PR 1908 is trying to resolve. Before PR 1908, `--packages` will pull the lib iceberg-core and its transitive dependencies, which takes a different path then the `--jar` option, as with the `--jar` option, the Polaris Spark client only needs the Iceberg-spark-runtime dependency. With PR 1908, we removed the dependency of iceberg-core, and only depends on Iceberg-spark-runtime[1], which is consistent with the `--jar` option. 1. https://github.com/apache/polaris/pull/1908/files#r2159414955 Yufei On Fri, Jun 20, 2025 at 10:08 AM Dmitri Bourlatchkov <di...@apache.org> wrote: > I definitely agree that we should resolve this issue for 1.0. > > ... but I'm not sure that running different code with --jar and --packages > is a good idea, even if the differences are only in references to shaded > classes. > > If one works without shading, why can the other not work without shading? > > Thanks, > Dmitri. > > On Fri, Jun 20, 2025 at 12:52 PM yun zou <yunzou.colost...@gmail.com> > wrote: > > > As for the following point > > I believe that regardless of the method of including the Client into > Spark > > runtime, the code has to be exactly the same.... and I doubt it is the > same > > now. WDYT? > > > > The code included in the jar for Spark Client is different now with the > > change, because it > > now uses a class in a different package, even though they do the same > > thing. However, > > I think it is a good change, it simplifies our dependency and avoids > > potential compatibility issue > > due to the shading of iceberg-spark-runtime. I definitely agree we should > > also include this also in 1.0. > > > > Best Regards, > > Yun > > > > On Fri, Jun 20, 2025 at 9:47 AM yun zou <yunzou.colost...@gmail.com> > > wrote: > > > > > > > > *-- What is the maven artifact that Spark can automatically pull > > > (via--packages)* > > > > > > Our spark client pulls the following: > > > > > > org.apache.polaris#polaris-spark-3.5_2.12 > > > > > > org.apache.polaris#polaris-core > > > > > > org.apache.polaris#polaris-api-management-model > > > > > > org.apache.iceberg#iceberg-spark-runtime-3.5_2.12 > > > > > > > > > Prior to the change, it also pulled iceberg-core and avro 1.20.0. > > > > > > > > > *-- Does that artifact use shaded dependencies* > > > > > > Any usage of classes from iceberg-spark-runtime uses the shaded > libraries > > > shipped along with the artifacts. > > > > > > > > > > > > *-- Does that artifact depend on the Iceberg Spark bundle?* > > > > > > If you are referring to our spark client, it depends on > > iceberg-spark-runtime, > > > not other bundles. > > > > > > > > > > > > *-- Is the _code_ running in Spark the same when the Polaris Spark > Client > > > ispulled via --packages and via --jars?* > > > > > > > > > yes, the jar and package will use the same code, where the jar simply > > > packs everything > > > > > > for the user and there is no need to download any other dependency. > > > > > > > > > Best Regards, > > > > > > Yun > > > > > > > > > > > > On Fri, Jun 20, 2025 at 9:18 AM Dmitri Bourlatchkov <di...@apache.org> > > > wrote: > > > > > >> Some questions for clarification: > > >> > > >> * What is the maven artifact that Spark can automatically pull (via > > >> --packages)? > > >> * Does that artifact use shaded dependencies? > > >> * Does that artifact depend on the Iceberg Spark bundle? > > >> * Is the _code_ running in Spark the same when the Polaris Spark > Client > > is > > >> pulled via --packages and via --jars? > > >> > > >> I know I could have figured that out from code, but I'm asking here > > >> because > > >> I think we may need to review our approach to publishing these > > artifacts. > > >> > > >> I believe that regardless of the method of including the Client into > > Spark > > >> runtime, the code has to be exactly the same.... and I doubt it is the > > >> same > > >> now. WDYT? > > >> > > >> Thanks, > > >> Dmitri. > > >> > > >> > > >> On Fri, Jun 20, 2025 at 10:15 AM Dmitri Bourlatchkov < > di...@apache.org> > > >> wrote: > > >> > > >> > Hi All, > > >> > > > >> > Re: PR [1908] let's use this thread to clarify the problems we're > > trying > > >> > to solve and options for solutions. > > >> > > > >> > As for me, it looks like some refactoring in the way the Spark > Client > > is > > >> > built and published may be needed. > > >> > > > >> > I think it makes sense to clarify this before 1.0 to avoid changes > to > > >> > Maven coordinates right after 1.0 > > >> > > > >> > [1908] https://github.com/apache/polaris/pull/1908 > > >> > > > >> > Thanks, > > >> > Dmitri. > > >> > > > >> > > > >> > > > > > >
