Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/2477#issuecomment-58454665
Thanks for including me, @JoshRosen!
I agree with @markhamstra here.
If I *had* to make this work, I wouldn't do the symlink. In fact, I'd
update compute-classpath.sh, to, in case of a RELEASE, look under $FWDIR/lib
AND $FWDIR/jars for the assembly jar. Symlink may cause some pain down the
road, so I'd prefer a backwards compatible and simple change in the
compute-classpath script.
I am not an expert on how the jdeb plugin is being used in Spark, but I
downloaded the latest binary tarball of spark and there is a "lib" directory
there. I can see jdeb taking the contents of lib directory and essentially
putting them under jars directory but I think it's a slippery slope to create
the symlink. If somehow a lib directory shows up under /usr/share/spark/, this
symlink creation would fail.
However, I also want to take this opportunity to raise a meta point.
Hadoop, and other projects when they were just starting, took the route of
bundling their own rpm/deb packaging code along with the project. They soon
realized that it was very hard to maintain that since the bread and butter of
the project is writing good, solid distributed software and not figuring out
why /var/run is a tempfs on latest Operating Systems. This is where Apache
Bigtop was born out of - to house the packaging for all projects of Hadoop
ecosystem and to integrate them with each other. Consequently, Apache Bigtop
packaging is considered the de-facto packaging for Hadoop ecosystem projects.
And, Spark is a part of it, has been for quite a while. So, while it's your
call at the end of the day, I'd strongly encourage you to separate the
packaging concern to Apache Bigtop since I think it's unreasonable to expect a
development project to accommodate the needs of all Operating Systems and
packaging stan
dards.
In interest of full-disclosure, I am a committer on the Apache Bigtop
project.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
