Hi Mats Rydberg, Although this dependency "org.opencypher:okapi-shade.okapi" was added into spark, but Xiangrui raised two concerns (see above mail) about it, so we'd better rethink on this and consider whether this is a good choice, so I call this vote.
Thanks! On Tue, Oct 15, 2019 at 10:56 PM Mats Rydberg <m...@neo4j.org.invalid> wrote: > Hello Weichen, community > > I'm sorry, I'm feeling a little bit confused about this vote. Is this > about the PR (https://github.com/apache/spark/pull/24490) that was merged > in early June and introduced the spark-graph module including the > okapi-shade dependency? > > Regarding the okapi-shade dependency which was developed as part of the > above PR work, some advice was offered by Scala experts at TripleQuote > which helped find a satisfactory solution. The shading mechanism used is > standard and very comparable to a Java library shading solution. > > The PR you link (https://github.com/apache/spark/pull/24297) is not meant > for merging. It is just a proof-of-concept branch containing a full > implementation of the system, which is kept up-to-date with the API > discussion on the currently proposed PR: > https://github.com/apache/spark/pull/24851. > > Thank you > Mats > > > On Tue, Oct 15, 2019 at 10:38 AM Weichen Xu <weichen...@databricks.com> > wrote: > >> Hi everyone, >> >> I'd like to call a new vote on the issue: should we add dependency >> "org.opencypher:okapi-shade.okapi" into spark ? The issue background is: >> >> Spark is going to add a big feature "Spark Graph", the prototypical >> implementation is here >> https://github.com/apache/spark/pull/24297 >> which will introduce dependency org.opencypher:okapi-shade.okapi >> <https://github.com/opencypher/morpheus/blob/master/okapi-shade/build.gradle> >> >> Xiangrui already mentioned 2 concerns on this dependency change: >> >>> On the technical side, my main concern is the runtime dependency on >>> org.opencypher:okapi-shade.okapi depends on several Scala libraries. We >>> came out with the solution to shade a few Scala libraries to avoid >>> pollution. However, I'm not super confident that the approach is >>> sustainable for two reasons: a) there exists no proper shading libraries >>> for Scala, 2) We will have to wait for upgrades from those Scala libraries >>> before we can upgrade Spark to use a newer Scala version. So it would be >>> great if some Scala experts can help review the current implementation and >>> help assess the risk. >> >> >> So let's discuss and vote whether this is a good choice. >> Before this spark graph feature to get into spark ASAP, this issue should >> be resolved first. >> >> This vote is open until next Tuseday (Oct. 22). >> >> [ ] +1: Accept the proposal >> [ ] +0 >> [ ] -1: I don't think this is a good idea because ... >> >> Thank you! >> >> Weichen >> >>
