Re: Status of sbt package
Den 27.09.2019 10:15, skrev mer...@debian.org: jline v2.14.6 is in Debian as libjline2-java, wouldn't this work? FYI I did a test of using jline 2.14.6 when running scala 2.12.0-M1 repl, and it failed with NoSuchMethodError. Downloading and compiling jline 2.14.1, on the other hand, did work. Interrestingly enough scala 2.11.12 uses jline 2.14.3, while scala 2.12.0 uses jline 2.14.1. A downgrade! All this is of course a different error than what I get while compiling, so I will investigate further the compile error. Thomas
Re: Status of sbt package
Hi Andrius, Den 27.09.2019 10:15, skrev mer...@debian.org: I have managed to get around Scala FTBFS problem [1] for now, will continue looking into your script. [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=941211 Are you trying to compile scala 2.11.12 from the source debian package or from the scalateams 2.11.12 src tar ball? Are you doing that to use it as the base to compile 2.12.0-M1, to be able to compile sbt or just to get expereience with build script build-2.12.0-M1.sh? The bug report mentioned maven? I am nort familiar with the build procedure described, are you using maven to start the ant build script? Or have I got it all wrong? jline v2.14.6 is in Debian as libjline2-java, wouldn't this work? I experimented with using the jline from /usr/share/scala/lib, but it did not work, I think the problem is not jline in it self, but rather that the bootclasspath is different in the debian package than the local installation of scala 2.11.12 I used when writing the initial script. By the way, how are the jars under $BUILDMODULEPATH built? The short answer is its the release library jars that comes with the scala release of 2.11.12, so its precompiled. Also some of the jars I just copied from the release tarball to be able to test if it I can compile a test scala class and run it. Specifically jline, scala-xml, config and a few others. The long answer is, in two parts (I appologise if this is obvious...) 1- I downloaded from efpl the release of 2.11.12 and used that as the compiler for the build. So when setting up the booclasspath I had to control which jars were included at the different stages of the compilation script. The next answer explains why. 2- Since we are compiling a compiler, we need to incrementally exchange the old version of the scala jar files with the newly compiled version, otherwise the new scala version wont run (abstract method error). This means overriding the complete bootclasspath for the jvm, not just scala booclasspath. So the strategy is simple, but technically confusing at times, by manually creating a new and complete javabootclasspath: - get all jars from the jre, they are put last on the bootCP - get all jars from the scala libs, they are put second to last on the bootCP - first use all this to build the first module of the new scala version: scala-library - replace old scala-library jar (from 2.11.12) with newly compiled library-jar (2.12.0-M1) on the bootCP (put first on bootCP to guarantee shadowing of the old version) - as the modules are built, keep replacing the old with the new, in the order of asm, forkjoin, library, reflect, compiler, scalap, repl, interactive The jars the script produces ar not package exactly the same as the efpl release jars are, so that might be a bit configusing Btw, maybe we should start a new thread about the scala compilation stuff so its easier for all to see what the discussions are actually about?
Re: Status of sbt package
Hi Thomas, I have managed to get around Scala FTBFS problem [1] for now, will continue looking into your script. [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=941211 On 2019-09-27 02:07, Thomas Finneid wrote: > 1- The scala library depenencies are hardcoded, so it doesnt find the > correct version of them, Still it compiles, which is wierd, but wont > run, mostlikely because of an AbstractMethodException or something > similar. Its a bootclasspath issue again. > 2- The external dependencies are not met, in this case jline. jline v2.14.6 is in Debian as libjline2-java, wouldn't this work? By the way, how are the jars under $BUILDMODULEPATH built? Best wishes, Andrius
Re: Status of sbt package
Den 26.09.2019 21:29, skrev Thomas Finneid: I am currently working on making the script work in debian unstable, using system installed packages for jdk, scala 2.11.12, ant, jline etc. I will probably have something workign by the end of the evening. Hi again, I have updated the scala build script so it compiles on Debian unstable now. There are a couple of errors, so it does not run to completion yet, but most of it compiles. The two major errors are 1- The scala library depenencies are hardcoded, so it doesnt find the correct version of them, Still it compiles, which is wierd, but wont run, mostlikely because of an AbstractMethodException or something similar. Its a bootclasspath issue again. 2- The external dependencies are not met, in this case jline. So... getting there... Thomas
Re: Status of sbt package
On 26.09.2019 16:54, mer...@debian.org wrote: I have cloned your repo and started playing around. AFAIK, your build script requires jars built from earlier version of Scala (akka-actor.jar and friends). Thus I've tried building Scala and ran into a FTBFS [1]. I will have to figure it out before moving to Scala 2.12.0-M1. [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=941211 Yes I used 2.11.12 as the build tool to build 2.12.0-M1. This is the way the Scala team has asked it be done, to do it properly. The script was initially created in Ubuntu, so I downloaed 2.11.12 and installed it manually into a separate directory, then set the PATH to point to that. There is a lot of hardcoding of paths and files in the script. I am currently working on making the script work in debian unstable, using system installed packages for jdk, scala 2.11.12, ant, jline etc. I will probably have something workign by the end of the evening. However, this strategy is for the time being tedious due to sbt not migrating to testing. Otherwise it would sound really tempting. I agree. Thomas
Re: Status of sbt package
Hi Thomas, On 2019-09-25 23:03, Thomas Finneid wrote: > Sorry for the delay in answering, I'm on daddy-leave so time is > scarce. (and sorry for the long mail... :) ) Thank you for finding some time to contribute to packaging Scala/sbt! :) > I have some more good news :) After battling with my own and other > demons I have managed to create a shell script for compiling Scala > 2.12.0-M1. The experiences from that can probably be used to create a > similar script for Sbt. Fantastic! > You can have a look at what I have done at > > https://bitbucket.org/tfinneid/scala-build-process/src/default/build-2.12.0-M1.sh > I have cloned your repo and started playing around. AFAIK, your build script requires jars built from earlier version of Scala (akka-actor.jar and friends). Thus I've tried building Scala and ran into a FTBFS [1]. I will have to figure it out before moving to Scala 2.12.0-M1. [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=941211 > I think one of the bigger issues for creating a working sbt debian > package is: > - sbt downloads the projects correct sbt version and other > dependencies and compiles the compiler interface for the correct scala > version before making the project available on sbt command line > - Setting that up correctly for the installed package, might not be > as easy as one thinks. > - I experienced some problems with that when testing the existing > sbt package in debian on a minor test project I created. > - I found that project dependencies was only retrieved from local > files installed by the package, not from the network as it should > (this applies to the completed and installed binary package, not the > building of the package > - other than that sbt external build dependencies must either exist as > jar files in debian or be created as separate packages (or included as > a separate pre-compile step in the sbt build script. > > Also, check what version of the jvm Sbt was originally built with. > Scala 2.12 is built using Java 8 and hence only properly compatible > with that. Scala 2.12.10 (at least) and 2.13 supports jdk11, but its > still built with jdk8 (not target argument, but actual jdk8 release). Ack. > sbt 0.13 is required for Scala 2.12, while sbt 1.0 is required for > Scala 2.13 builds. I dont know if that is important information yet, > it depends on the strategy for moving forward. However, this strategy is for the time being tedious due to sbt not migrating to testing. Otherwise it would sound really tempting. Many thanks for sharing your scripts and experience! I will let you know as soon as I manage to build Scala 2.12.0-M1. Best wishes, Andrius
Re: Status of sbt package
Hi Andrius Sorry for the delay in answering, I'm on daddy-leave so time is scarce. (and sorry for the long mail... :) ) I have some more good news :) After battling with my own and other demons I have managed to create a shell script for compiling Scala 2.12.0-M1. The experiences from that can probably be used to create a similar script for Sbt. The original build file (sbt) for Scala is miles long and has lots of stuff not needed for creating a debian package. I suspect the same is true for Sbt. The original scripts have all sorts of bootstrapping, infrastructure, test and other administrative or maintenance parts, which is not needed for our purpose. All that is taken care of upstream, we just need to build and ship it. My strategy was reverse engineering of the build process, using the scala version that exists in Debian (2.11.12) and verbose mode in sbt to study the output from a compilation or packaging task. I know a different strategy was used for Sbt, but I think this is the more viable solution. A Scala release build is somewhat different from a Sbt release build. Since I was compiling the compiler, I needed to override the bootclasspath so that the new project modules replaced the older modules on the bootclasspath. Thats not a problem for sbt, but there still might be some interresting classpath issues when compiling sbt. You can have a look at what I have done at https://bitbucket.org/tfinneid/scala-build-process/src/default/build-2.12.0-M1.sh (The scripts is a very naive version, but it actually caused me more headache trying to creating a more automatic version of the script while, at the same time, trying to figure out the correct Scala build process and command arguments to get a working scala release) I think one of the bigger issues for creating a working sbt debian package is: - sbt downloads the projects correct sbt version and other dependencies and compiles the compiler interface for the correct scala version before making the project available on sbt command line - Setting that up correctly for the installed package, might not be as easy as one thinks. - I experienced some problems with that when testing the existing sbt package in debian on a minor test project I created. - I found that project dependencies was only retrieved from local files installed by the package, not from the network as it should (this applies to the completed and installed binary package, not the building of the package - other than that sbt external build dependencies must either exist as jar files in debian or be created as separate packages (or included as a separate pre-compile step in the sbt build script. Also, check what version of the jvm Sbt was originally built with. Scala 2.12 is built using Java 8 and hence only properly compatible with that. Scala 2.12.10 (at least) and 2.13 supports jdk11, but its still built with jdk8 (not target argument, but actual jdk8 release). FYI sbt 0.13 is required for Scala 2.12, while sbt 1.0 is required for Scala 2.13 builds. I dont know if that is important information yet, it depends on the strategy for moving forward. Hope this helps. And I love to hear what you find and share my experiences. Thomas On 10.09.2019 17:56, mer...@debian.org wrote: Hi Frédéric and Thomas, Sorry for the delay. I must admit I wasn't aware the dogfooding problem is so difficult to avoid. On 2019-08-17 07:53, Thomas Finneid wrote: The good news is that after talking with some of the maintainers of Sbt and Scala, they would be very interrested in getting a script that could build Sbt or Scala from command line, using just Java and similr tools. This solution sounds the best. Do you know whether the upstream would be willing to help with writing and maintaining such a script? Building sbt from 0.13.13 incrementally would be an option too, should the previous one fail. However, to build sbt incrementally one would probably need to demote its dependencies to make them compatible with this ancient version of source. Not sure where to start, though. Thanks for sharing your experience. Best, Andrius
Re: Status of sbt package
Hi Frédéric and Thomas, Sorry for the delay. I must admit I wasn't aware the dogfooding problem is so difficult to avoid. On 2019-08-17 07:53, Thomas Finneid wrote: > The good news is that after talking with some of the maintainers of > Sbt and Scala, they would be very interrested in getting a script that > could build Sbt or Scala from command line, using just Java and similr > tools. This solution sounds the best. Do you know whether the upstream would be willing to help with writing and maintaining such a script? Building sbt from 0.13.13 incrementally would be an option too, should the previous one fail. However, to build sbt incrementally one would probably need to demote its dependencies to make them compatible with this ancient version of source. Not sure where to start, though. Thanks for sharing your experience. Best, Andrius
Re: Status of sbt package
Den 14.08.2019 11:54, skrev Frédéric Bonnard: But for sbt, as far as I remember, you cannot just javac * and scalac * (as for the other simple sbt- packages : see d/rules of sbt-launcher-interface, sbt-serialization etc), because pieces of the source code is generated during the build process and I didn't find time to fake that source generation without the embedded sbt, with only javac/scalac. Currently, sbt in the archive is 0.13.13 and upstream seems to propose 1.3 . So, is it more easy to try to build 1.3 from scratch or do we need to pursue the effort on 0.13.13 to be able to then build 1.3 ... only digging more can tell :) Hi Frederic and Andrius I think getting 1.3 working in debian is far more valuable to all parties, Debian, Scala and the Sbt projects. 0.13 is old and not required by the latest version of the projects. I looked into building Sbt and Scala half a year ago, trying to get to grips with how to build on the Debian build process, but found the dependencies to be circular. As it is, Sbt requires Scala and Scala requires Sbt to compile. In addition it requires the previous version of the spesific build project. I.e to build Scala 2.12.8, one needs Scala 2.12.7 and so on. The same for Sbt. So as far as I understand, and remember, in Debian you would first need a proper debian build of Scala, to get any where with building Sbt. The good news is that after talking with some of the maintainers of Sbt and Scala, they would be very interrested in getting a script that could build Sbt or Scala from command line, using just Java and similr tools. The propper build process for both Scala and Sbt contains a lot of project and QA activites that we dont need for a debian build. That is, before any PR's are merged, the projects own build process does the quality checks required. So the only quality checks a Debian build would require is, to make sure it builds and runs properly after a Debian build and adheres to all Debian packaging rules. That allows us to simplify the build process for both Scala and Sbt, for Debian purposes. Thats how far I got, but I would love to help and continue working on this. Andrius, whats your thoughts or plans on how to build Sbt? Regards Thomas
Re: Status of sbt package
Hi, On Fri, 9 Aug 2019 18:25:19 +0300, mer...@debian.org wrote: > On 2019-08-08 17:48, Emmanuel Bourg wrote: > > I don't think there is a repository yet. I've just created an empty one > > on Salsa if that helps. > > Thanks - I'll push once I have something. the point is that the current packaging of sbt, uses loads of embedded binaries to bootstrap sbt, which is not very Vcs friendly. Emmanuel tricked several sbt runtime dependencies to have them build without those embedded binaries, which is great with only javac and scalac because the build process is simple enough. But for sbt, as far as I remember, you cannot just javac * and scalac * (as for the other simple sbt- packages : see d/rules of sbt-launcher-interface, sbt-serialization etc), because pieces of the source code is generated during the build process and I didn't find time to fake that source generation without the embedded sbt, with only javac/scalac. Currently, sbt in the archive is 0.13.13 and upstream seems to propose 1.3 . So, is it more easy to try to build 1.3 from scratch or do we need to pursue the effort on 0.13.13 to be able to then build 1.3 ... only digging more can tell :) F. pgpOGr_Ol8VTb.pgp Description: PGP signature
Re: Status of sbt package
On 2019-08-08 17:48, Emmanuel Bourg wrote: > I don't think there is a repository yet. I've just created an empty one > on Salsa if that helps. Thanks - I'll push once I have something. Best, Andrius
Re: Status of sbt package
Le 08/08/2019 à 12:37, mer...@debian.org a écrit : > Thanks, I'll give it a look. By the way, the source package does not > have VCS-* fields, neither it is on Salsa. Do you know where its > packaging repository is? I don't think there is a repository yet. I've just created an empty one on Salsa if that helps. Emmanuel Bourg
Re: Status of sbt package
Hi Emmanuel, On 2019-08-08 10:33, Emmanuel Bourg wrote: > sbt is in sid but hasn't been bootstrapped properly yet, it still uses > libraries not built from sources. Nobody is actively working on it, feel > free to give it a try. Thanks, I'll give it a look. By the way, the source package does not have VCS-* fields, neither it is on Salsa. Do you know where its packaging repository is? Best wishes, Andrius signature.asc Description: OpenPGP digital signature
Re: Status of sbt package
Hi Andrius, sbt is in sid but hasn't been bootstrapped properly yet, it still uses libraries not built from sources. Nobody is actively working on it, feel free to give it a try. Emmanuel Bourg Le 07/08/2019 à 13:32, mer...@debian.org a écrit : > Hello, > > I am writing regarding the status of sbt Debian package. Version 0.13.13 > is in sid, but it had never entered testing. Version 1.3.0-RC3 has > already been released, are there any plans to update the package? If > not, I would be willing to try my luck. > > Best wishes, > Andrius
