J maintainers: I attached a patch to add CMake configurations to
- build j, jconsole, jnative, and tsdll in Release and Debug - run unit tests It is tested on Windows and Linux, inside and outside Visual Studio 2019, with Clang and GCC. It should work on Mac as well. The rest are the details of my proposal. # What are the problems I'm trying to solve? 1. We are maintaining too many build configurations. The J project currently has at least 5 sets of build configurations using 3 different build systems. The confusion that added up significantly increases the bar of consumption and contribution. 2. It adds potential problems to develop in one build system and release in another. Currently, for people who develop in Visual Studio (MSBuild), they will find that the official release is done with nmake, using a different set of flags and a different linker. Making the program to be warning-free, error-free for both build configurations are a waste of developers' energy. 3. The existing build configurations are too slow. The complete J build consists of <180 objects, which should be done in seconds, on all platforms, with Ninja. It is pointless to build a high-performance application with 6% of your CPU computational power. # What can solve these problems? So we have 5, why does my patch add another one? Since everybody who approaches you will say "it is a part of a bigger plan" so I'm not going to say that. The patch I propose already does a lot, which I will explain later. I need J maintainers' input before I jump ahead and add all possible combinations that you don't use. The idea is that we need only one. Not a build system, but a configuration for a build system generator, that runs on all platforms, can target all platforms, supports all toolchains, and can be load in all modern IDEs. That is CMake. 1. CMake runs on all platforms that we may develop on. Windows, Linux, OS X. When it generates builds for Ninja, ninja runs on these platforms as well. 2. CMake can target all platforms. The idea is that CMake allows the users to prepare their builds in a descriptive scripting language for all possible target platforms. And then, when generating the build configuration, you use a "toolchain file" to describe the toolchain and the platform. For Android, it is as simple as cmake -B build -DCMAKE_TOOLCHAIN_FILE=path-to-ndroid-ndk-r21d\build\cmake\android.toolchain.cmake The NDK website says: Android Studio's default build tool to compile native libraries is CMake. [...] if you are creating a new native library, you should use CMake. 3. CMake supports all toolchains. All versions of Clang, GCC, Clang-cl, MSVC, Intel -- they are nothing to CMake. Assemblers, Windows Resource Compiler, building shared libraries cross platforms -- as simple as adding the file name. You can even mix C and CUDA in the same project. 4. Can be load in all modern IDEs Visual Studio and CLion natively support CMake projects (generating Ninja builds). CMake can also generate build configurations in MSBuild and Xcode projects if you really like those. Visual Studio Code relies on CMake as well (Microsoft owns the CMake Tools extension). CMake went far beyond that. It supports everything that can help to create an out-of-box experience for developers. Here are some examples: 1. Generate code. CMake can generate source files from templates and allows you to describe that task as a "target" to let other tasks depend on. We can eliminate the need of updating version files with this. 2. Downloading and decompress files We can instruct the build system to obtain a copy of nasm binary, build OpenSSL assembly code on Windows instead of carrying object files over. 3. Run unit tests in parallel CMake has a framework of incorporating unit tests into the build. With my patch, you can run 406 test suites with `ninja -C <the-build-directory> test` or "Test" -> "Run CTests for j" from Visual Studio menu. Those finishes in about 150 seconds. You can read https://cmake.org/cmake/help/latest/manual/ctest.1.html#options to learn how to reduce that time to <60s. Need to rename some temporary file names to avoid collisions though. In a word, CMake can do everything and solve all the problems I listed above. # How to use my patch? You are welcome to apply my patch on top of 8cf651b6 and select "Open" -> "CMake..." or "Folder..." from the menu in Visual Studio 2019 and see what happens next. If you are using the command-line under Linux, please install cmake >= 3.17 and ninja. If you cannot find a recent software repository, you can open an Anaconda environment and install them there. And then, execute cmake -G "Ninja Multi-Config" -B build under the jsource root directory. The last argument "build" is for specifying the build directory. It can be anything but I will use "build" for the rest of the article. The above command generates a Ninja multi-configuration build. Now running ninja -C build will build j, jconsole in Debug, and prepare for unit tests. If we found JDK we will also build jnative. The executable is now usable with ./build/jsrc/Debug/jconsole It loads jlibrary under your jsource repository. T run unit tests given the Debug executable and libraries ninja -C build test Do the Release build ninja -C build -f build-Release.ninja Run unit tests with the Release executable and libraries: ninja -C build -f build-Release.ninja test Note that the Release artifacts and Debug artifacts can coexist. The Release executable is at ./build/jsrc/Release/jconsole Note that you don't need to re-run cmake. If you made changes to any source file, you do an incremental build with ninja. If you made changes to CMakeLists.txt, run ninja as well. Ninja knows when does it need to reconfigure. If you totally messed up, just rm -fr build. You can use command-line under Windows as well in "Developer Command Prompt for VS 2019," except that you need to specify compilers: cmake -B build -G "Ninja Multi-Config" -DCMAKE_C_COMPILER=clang-cl -DCMAKE_CXX_COMPILER=clang-cl Under Linux, you can switch compilers to Clang with cmake -B build -G "Ninja Multi-Config" -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ # Where am I and what comes next? Currently, my patch only builds avx2. I will make it work with other ISAs but I need a complete list of all the target platforms we want to support so that I can come up with a design. I suggest dropping 32-bit support, require avx2 on x86 architecture and >= arm64-v8a on ARM. After completing the ISA that I can test Android support. My patch looks for OpenMP and builds with it if it can. On Windows, I instructed the build system to automatically copy libomp.dll from clang-cl installation. I did not give you an option to build without OpenMP (6% CPU usage is not high performance). I hope people don't disagree with this but it's something I bring up for discussion. Similarly, it always tries to build with readline; linenoise on Windows. I did not provide an option to turn this off. My build configuration should be able to support building JEXEServer though. I haven't tested this, but the idea is that if you use `add_subdirectory` to add jsource repo as a subproject, I instruct CMake to switch off dynamic libraries and anything other than libj, so that you can link to a static libj with `target_link_libraries`. `add_subdirectory` is a common practice to CMake projects. After the new build configuration reaches a stage that it can replace all the others, we should do a cleanup. Not only those configurations but also the historical artifacts. Then we may be able to do some improvements using code generation etc. In general, automate the parts of the build process that we couldn't automate before. # Summary That's my plan. It would be nice if we can schedule a Zoom meeting so that I can demonstrate these and answer questions. By any chance, I would like to hear your thoughts. Thanks, -- Zhihao ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
