The mailing list software deployed for the jsoftware forums filters out attachments.
You should probably submit a pull request against https://github.com/jsoftware/jsource if you want anyone to see this. (Though, worst case, including it as an inline document might work, if none of the lines are too long.) Thanks, -- Raul On Tue, Nov 10, 2020 at 4:04 AM 'Zhihao Yuan' via Source <[email protected]> wrote: > > J maintainers: > > I attached a patch to add CMake configurations to > > - build j, jconsole, jnative, and tsdll in Release and Debug > - run unit tests > > It is tested on Windows and Linux, inside and outside > Visual Studio 2019, with Clang and GCC. It should > work on Mac as well. The rest are the details of my > proposal. > > # What are the problems I'm trying to solve? > > 1. We are maintaining too many build configurations. > > The J project currently has at least 5 sets of build > configurations using 3 different build systems. > The confusion that added up significantly increases > the bar of consumption and contribution. > > 2. It adds potential problems to develop in one build > system and release in another. > > Currently, for people who develop in Visual Studio > (MSBuild), they will find that the official release is > done with nmake, using a different set of flags and > a different linker. Making the program to be > warning-free, error-free for both build configurations > are a waste of developers' energy. > > 3. The existing build configurations are too slow. > > The complete J build consists of <180 objects, which > should be done in seconds, on all platforms, with Ninja. > It is pointless to build a high-performance application > with 6% of your CPU computational power. > > # What can solve these problems? > > So we have 5, why does my patch add another one? > Since everybody who approaches you will say "it is > a part of a bigger plan" so I'm not going to say that. > The patch I propose already does a lot, which I will > explain later. I need J maintainers' input before I > jump ahead and add all possible combinations > that you don't use. > > The idea is that we need only one. Not a build system, > but a configuration for a build system generator, that > runs on all platforms, can target all platforms, supports > all toolchains, and can be load in all modern IDEs. > That is CMake. > > 1. CMake runs on all platforms that we may develop on. > Windows, Linux, OS X. When it generates builds for > Ninja, ninja runs on these platforms as well. > > 2. CMake can target all platforms. The idea is that > CMake allows the users to prepare their builds > in a descriptive scripting language for all possible > target platforms. And then, when generating the > build configuration, you use a "toolchain file" to > describe the toolchain and the platform. For > Android, it is as simple as > > cmake -B build > -DCMAKE_TOOLCHAIN_FILE=path-to-ndroid-ndk-r21d\build\cmake\android.toolchain.cmake > > The NDK website says: > > Android Studio's default build tool to compile > > native libraries is CMake. [...] if you are creating > > a new native library, you should use CMake. > > > 3. CMake supports all toolchains. > > All versions of Clang, GCC, Clang-cl, MSVC, Intel > -- they are nothing to CMake. Assemblers, Windows > Resource Compiler, building shared libraries cross > platforms -- as simple as adding the file name. You > can even mix C and CUDA in the same project. > > 4. Can be load in all modern IDEs > > Visual Studio and CLion natively support CMake > projects (generating Ninja builds). CMake can also > generate build configurations in MSBuild > and Xcode projects if you really like those. > Visual Studio Code relies on CMake as well > (Microsoft owns the CMake Tools extension). > > CMake went far beyond that. It supports everything > that can help to create an out-of-box experience for > developers. Here are some examples: > > 1. Generate code. > > CMake can generate source files from templates > and allows you to describe that task as a "target" > to let other tasks depend on. We can eliminate > the need of updating version files with this. > > 2. Downloading and decompress files > > We can instruct the build system to obtain a copy > of nasm binary, build OpenSSL assembly code > on Windows instead of carrying object files over. > > 3. Run unit tests in parallel > > CMake has a framework of incorporating unit tests > into the build. With my patch, you can run 406 > test suites with `ninja -C <the-build-directory> test` or > "Test" -> "Run CTests for j" from Visual Studio > menu. Those finishes in about 150 seconds. > You can read > https://cmake.org/cmake/help/latest/manual/ctest.1.html#options > to learn how to reduce that time to <60s. Need > to rename some temporary file names to avoid > collisions though. > > In a word, CMake can do everything and solve > all the problems I listed above. > > # How to use my patch? > > You are welcome to apply my patch on top of > 8cf651b6 and select "Open" -> "CMake..." or > "Folder..." from the menu in Visual Studio > 2019 and see what happens next. > > If you are using the command-line under Linux, > please install cmake >= 3.17 and ninja. If you > cannot find a recent software repository, you > can open an Anaconda environment and install > them there. And then, execute > > cmake -G "Ninja Multi-Config" -B build > > under the jsource root directory. The last argument > "build" is for specifying the build directory. It can > be anything but I will use "build" for the rest of > the article. > > The above command generates a Ninja > multi-configuration build. Now running > > ninja -C build > > will build j, jconsole in Debug, and prepare for > unit tests. If we found JDK we will also build jnative. > The executable is now usable with > > ./build/jsrc/Debug/jconsole > > It loads jlibrary under your jsource repository. > > T run unit tests given the Debug executable and > libraries > > ninja -C build test > > Do the Release build > > ninja -C build -f build-Release.ninja > > Run unit tests with the Release executable and > libraries: > > ninja -C build -f build-Release.ninja test > > Note that the Release artifacts and Debug > artifacts can coexist. The Release executable > is at > > ./build/jsrc/Release/jconsole > > Note that you don't need to re-run cmake. > If you made changes to any source file, > you do an incremental build with ninja. > If you made changes to CMakeLists.txt, > run ninja as well. Ninja knows when does > it need to reconfigure. If you totally messed > up, just rm -fr build. > > You can use command-line under Windows > as well in "Developer Command Prompt for > VS 2019," except that you need to specify > compilers: > > cmake -B build -G "Ninja Multi-Config" -DCMAKE_C_COMPILER=clang-cl > -DCMAKE_CXX_COMPILER=clang-cl > > Under Linux, you can switch compilers > to Clang with > > cmake -B build -G "Ninja Multi-Config" -DCMAKE_C_COMPILER=clang > -DCMAKE_CXX_COMPILER=clang++ > > # Where am I and what comes next? > > Currently, my patch only builds avx2. I will > make it work with other ISAs but I need a > complete list of all the target platforms we > want to support so that I can come up with > a design. I suggest dropping 32-bit support, > require avx2 on x86 architecture and > >= arm64-v8a on ARM. > > After completing the ISA that I can test > Android support. > > My patch looks for OpenMP and builds with > it if it can. On Windows, I instructed the > build system to automatically copy libomp.dll > from clang-cl installation. I did not give you > an option to build without OpenMP > (6% CPU usage is not high performance). > I hope people don't disagree with this but it's > something I bring up for discussion. > > Similarly, it always tries to build with readline; > linenoise on Windows. I did not provide > an option to turn this off. > > My build configuration should be able to > support building JEXEServer though. > I haven't tested this, but the idea is that > if you use `add_subdirectory` to add jsource > repo as a subproject, I instruct CMake > to switch off dynamic libraries and anything > other than libj, so that you can link to a > static libj with `target_link_libraries`. > `add_subdirectory` is a common practice > to CMake projects. > > After the new build configuration reaches a > stage that it can replace all the others, we > should do a cleanup. Not only those > configurations but also the historical artifacts. > > Then we may be able to do some > improvements using code generation etc. > In general, automate the parts of the build > process that we couldn't automate before. > > # Summary > > That's my plan. It would be nice if we can > schedule a Zoom meeting so that I can > demonstrate these and answer questions. > By any chance, I would like to hear your > thoughts. > > Thanks, > > -- > Zhihao > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
