AW: [DISCUSS] Refactor the way we're building/using Thrift
So, I’ve just merged the PR after updating it with the latest changes from master and the CI/CD gave it’s green light. So now it should be a lot simpler to build thrift and to manage new os/architecture/thrift-versions as we no longer have to manually copy hand-built executables to our second git-repo. Right now, all that would be needed, would be to run this command: ./mvnw clean deploy -P with-tools -pl :iotdb-tools-thrift It would build the assembly for the system the command is run on and deploy that in the Nexus snapshot repo, from where everyone can fetch it, without building it locally. As the tools are currently part of the build, we would need to deploy these convenience assemblies for each supported platform. I would therefore propose to move that part of the build out of the main project. We could simply add it to a separate git-reposiroty. There we would simply release that whenever we want to switch to a new Thrift version (which is probably going to happen very infrequently). This would simplify releasing even more. Chris Von: Christofer Dutz Datum: Dienstag, 1. August 2023 um 16:00 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift Ok … so after quite a while of refactoring and cleaning up and tweaking, I think my PR is ready for a review. Chris Von: Christofer Dutz Datum: Montag, 31. Juli 2023 um 10:00 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift And again sorry, turns out I selected the branches the wrong way around … this is the right PR: https://github.com/apache/iotdb/pull/10742 Chris Von: Christofer Dutz Datum: Montag, 31. Juli 2023 um 09:33 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift Hi Folks, So, I’ve literally spent all weekend working on my refactoring and must admit I’m quite happy with the results. However, did I decide to take the PR out of this repo and work on a fork in my GitHub account as I didn’t want to swamp you folks with build failure emails. So, I closed the old PR and opened a new one: https://github.com/apache/iotdb/pull/10741 Right now, I still have two TODOs that I would like to work on: * Make the cpp-client build again on Windows (AARCH64 and x86_64) (Currently struggling with it not detecting that it needs a different socket implementation) * Introduce an “iotdb-test-server” module, that integration tests use for running tests against a server: * The current usage of calling scripts in distribution introduces a logical cycle in Maven (Example: cpp-client needs distribution to run tests, distribution bundles cpp-client) * I had many build failures, as the IoTDB server instances are not shut down cleanly. I have noticed that we generally seem to have a number of flaky tests, that randomly fail … as I’ve been running dozens of full builds with integration tests on all supported platforms, I’ve had to re-run many builds to continue from the last failure. I’ve started compiling a list of these tests and I think we should fix them, as the CI should generally pass and not require an arbitrary number of re-runs. Chris Von: Christofer Dutz Datum: Samstag, 29. Juli 2023 um 16:22 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift Well it seems it’s not quite that simple. Because for example the CMakeList.txt references directories outside of the module itself. https://github.com/apache/iotdb/blob/master/iotdb-client/client-cpp/src/main/CMakeLists.txt If you wanted to build the client without having built thrift, this will not work. I’m currently working on making the thrift module sort of independent from the rest. The idea is to build the artifact once on every target platform and to make the produced artifacts available via Maven. Instead of just having one executable downloaded, I’d propose to have an archive containing the: * Thrift executable * Thrift runtime shared libraries * Thrift header files With this every module, would just unpack that archive and use that locally. This removes any circular dependencies. However, this module sort of doesn’t fit nicely into the build and it will cause problems. It will always just build thrift for the one platform the RM is currently using. So, I would propose to move this tiny project into the other git repository, that we currently use to serve the pre-compiled binaries. There whenever we need a new OS, CPU Architecture or Thrift version, we update that module and release it (and stage new thrift assembies in Nexus) This would eliminate all complexities in the build. Chris Von: Xiangdong Huang Datum: Samstag, 29. Juli 2023 um 08:37 An: dev@iotdb.apache.org Betreff: Re: [DISCUSS] Refactor the way we're building/using Thrift Hi Chris, > I would like to move the compile-tools directory into the root of the project > and detach i
AW: [DISCUSS] Refactor the way we're building/using Thrift
Ok … so after quite a while of refactoring and cleaning up and tweaking, I think my PR is ready for a review. Chris Von: Christofer Dutz Datum: Montag, 31. Juli 2023 um 10:00 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift And again sorry, turns out I selected the branches the wrong way around … this is the right PR: https://github.com/apache/iotdb/pull/10742 Chris Von: Christofer Dutz Datum: Montag, 31. Juli 2023 um 09:33 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift Hi Folks, So, I’ve literally spent all weekend working on my refactoring and must admit I’m quite happy with the results. However, did I decide to take the PR out of this repo and work on a fork in my GitHub account as I didn’t want to swamp you folks with build failure emails. So, I closed the old PR and opened a new one: https://github.com/apache/iotdb/pull/10741 Right now, I still have two TODOs that I would like to work on: * Make the cpp-client build again on Windows (AARCH64 and x86_64) (Currently struggling with it not detecting that it needs a different socket implementation) * Introduce an “iotdb-test-server” module, that integration tests use for running tests against a server: * The current usage of calling scripts in distribution introduces a logical cycle in Maven (Example: cpp-client needs distribution to run tests, distribution bundles cpp-client) * I had many build failures, as the IoTDB server instances are not shut down cleanly. I have noticed that we generally seem to have a number of flaky tests, that randomly fail … as I’ve been running dozens of full builds with integration tests on all supported platforms, I’ve had to re-run many builds to continue from the last failure. I’ve started compiling a list of these tests and I think we should fix them, as the CI should generally pass and not require an arbitrary number of re-runs. Chris Von: Christofer Dutz Datum: Samstag, 29. Juli 2023 um 16:22 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift Well it seems it’s not quite that simple. Because for example the CMakeList.txt references directories outside of the module itself. https://github.com/apache/iotdb/blob/master/iotdb-client/client-cpp/src/main/CMakeLists.txt If you wanted to build the client without having built thrift, this will not work. I’m currently working on making the thrift module sort of independent from the rest. The idea is to build the artifact once on every target platform and to make the produced artifacts available via Maven. Instead of just having one executable downloaded, I’d propose to have an archive containing the: * Thrift executable * Thrift runtime shared libraries * Thrift header files With this every module, would just unpack that archive and use that locally. This removes any circular dependencies. However, this module sort of doesn’t fit nicely into the build and it will cause problems. It will always just build thrift for the one platform the RM is currently using. So, I would propose to move this tiny project into the other git repository, that we currently use to serve the pre-compiled binaries. There whenever we need a new OS, CPU Architecture or Thrift version, we update that module and release it (and stage new thrift assembies in Nexus) This would eliminate all complexities in the build. Chris Von: Xiangdong Huang Datum: Samstag, 29. Juli 2023 um 08:37 An: dev@iotdb.apache.org Betreff: Re: [DISCUSS] Refactor the way we're building/using Thrift Hi Chris, > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). I quickly check current compile-tool, now it is only used by client-cpp. So is it really necessary to move to the root folder? > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). Yes, I also agree to compile tools ourselves, rather than download from somewhere.. (and just run once for each operation system.) > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture +1. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Christofer Dutz 于2023年7月28日周五 20:42写道： > > Hi, > > So, I've been fighting to simplify our use of Thrift, as there were several > problems with how we do it today on my M2 Mac (Mostly however to an updated > Homebrew, which uses different paths). > > While going through the project, I noticed that everything in "protocol" > downloads a pre-built executable, which is simply downloaded and run this > doesn’
AW: [DISCUSS] Refactor the way we're building/using Thrift
And again sorry, turns out I selected the branches the wrong way around … this is the right PR: https://github.com/apache/iotdb/pull/10742 Chris Von: Christofer Dutz Datum: Montag, 31. Juli 2023 um 09:33 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift Hi Folks, So, I’ve literally spent all weekend working on my refactoring and must admit I’m quite happy with the results. However, did I decide to take the PR out of this repo and work on a fork in my GitHub account as I didn’t want to swamp you folks with build failure emails. So, I closed the old PR and opened a new one: https://github.com/apache/iotdb/pull/10741 Right now, I still have two TODOs that I would like to work on: * Make the cpp-client build again on Windows (AARCH64 and x86_64) (Currently struggling with it not detecting that it needs a different socket implementation) * Introduce an “iotdb-test-server” module, that integration tests use for running tests against a server: * The current usage of calling scripts in distribution introduces a logical cycle in Maven (Example: cpp-client needs distribution to run tests, distribution bundles cpp-client) * I had many build failures, as the IoTDB server instances are not shut down cleanly. I have noticed that we generally seem to have a number of flaky tests, that randomly fail … as I’ve been running dozens of full builds with integration tests on all supported platforms, I’ve had to re-run many builds to continue from the last failure. I’ve started compiling a list of these tests and I think we should fix them, as the CI should generally pass and not require an arbitrary number of re-runs. Chris Von: Christofer Dutz Datum: Samstag, 29. Juli 2023 um 16:22 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift Well it seems it’s not quite that simple. Because for example the CMakeList.txt references directories outside of the module itself. https://github.com/apache/iotdb/blob/master/iotdb-client/client-cpp/src/main/CMakeLists.txt If you wanted to build the client without having built thrift, this will not work. I’m currently working on making the thrift module sort of independent from the rest. The idea is to build the artifact once on every target platform and to make the produced artifacts available via Maven. Instead of just having one executable downloaded, I’d propose to have an archive containing the: * Thrift executable * Thrift runtime shared libraries * Thrift header files With this every module, would just unpack that archive and use that locally. This removes any circular dependencies. However, this module sort of doesn’t fit nicely into the build and it will cause problems. It will always just build thrift for the one platform the RM is currently using. So, I would propose to move this tiny project into the other git repository, that we currently use to serve the pre-compiled binaries. There whenever we need a new OS, CPU Architecture or Thrift version, we update that module and release it (and stage new thrift assembies in Nexus) This would eliminate all complexities in the build. Chris Von: Xiangdong Huang Datum: Samstag, 29. Juli 2023 um 08:37 An: dev@iotdb.apache.org Betreff: Re: [DISCUSS] Refactor the way we're building/using Thrift Hi Chris, > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). I quickly check current compile-tool, now it is only used by client-cpp. So is it really necessary to move to the root folder? > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). Yes, I also agree to compile tools ourselves, rather than download from somewhere.. (and just run once for each operation system.) > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture +1. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Christofer Dutz 于2023年7月28日周五 20:42写道： > > Hi, > > So, I've been fighting to simplify our use of Thrift, as there were several > problems with how we do it today on my M2 Mac (Mostly however to an updated > Homebrew, which uses different paths). > > While going through the project, I noticed that everything in "protocol" > downloads a pre-built executable, which is simply downloaded and run this > doesn’t seem to run on Arm64 (new M1 and M2 Macs). > > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). > > This only needs
AW: [DISCUSS] Refactor the way we're building/using Thrift
Hi Folks, So, I’ve literally spent all weekend working on my refactoring and must admit I’m quite happy with the results. However, did I decide to take the PR out of this repo and work on a fork in my GitHub account as I didn’t want to swamp you folks with build failure emails. So, I closed the old PR and opened a new one: https://github.com/apache/iotdb/pull/10741 Right now, I still have two TODOs that I would like to work on: * Make the cpp-client build again on Windows (AARCH64 and x86_64) (Currently struggling with it not detecting that it needs a different socket implementation) * Introduce an “iotdb-test-server” module, that integration tests use for running tests against a server: * The current usage of calling scripts in distribution introduces a logical cycle in Maven (Example: cpp-client needs distribution to run tests, distribution bundles cpp-client) * I had many build failures, as the IoTDB server instances are not shut down cleanly. I have noticed that we generally seem to have a number of flaky tests, that randomly fail … as I’ve been running dozens of full builds with integration tests on all supported platforms, I’ve had to re-run many builds to continue from the last failure. I’ve started compiling a list of these tests and I think we should fix them, as the CI should generally pass and not require an arbitrary number of re-runs. Chris Von: Christofer Dutz Datum: Samstag, 29. Juli 2023 um 16:22 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift Well it seems it’s not quite that simple. Because for example the CMakeList.txt references directories outside of the module itself. https://github.com/apache/iotdb/blob/master/iotdb-client/client-cpp/src/main/CMakeLists.txt If you wanted to build the client without having built thrift, this will not work. I’m currently working on making the thrift module sort of independent from the rest. The idea is to build the artifact once on every target platform and to make the produced artifacts available via Maven. Instead of just having one executable downloaded, I’d propose to have an archive containing the: * Thrift executable * Thrift runtime shared libraries * Thrift header files With this every module, would just unpack that archive and use that locally. This removes any circular dependencies. However, this module sort of doesn’t fit nicely into the build and it will cause problems. It will always just build thrift for the one platform the RM is currently using. So, I would propose to move this tiny project into the other git repository, that we currently use to serve the pre-compiled binaries. There whenever we need a new OS, CPU Architecture or Thrift version, we update that module and release it (and stage new thrift assembies in Nexus) This would eliminate all complexities in the build. Chris Von: Xiangdong Huang Datum: Samstag, 29. Juli 2023 um 08:37 An: dev@iotdb.apache.org Betreff: Re: [DISCUSS] Refactor the way we're building/using Thrift Hi Chris, > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). I quickly check current compile-tool, now it is only used by client-cpp. So is it really necessary to move to the root folder? > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). Yes, I also agree to compile tools ourselves, rather than download from somewhere.. (and just run once for each operation system.) > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture +1. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Christofer Dutz 于2023年7月28日周五 20:42写道： > > Hi, > > So, I've been fighting to simplify our use of Thrift, as there were several > problems with how we do it today on my M2 Mac (Mostly however to an updated > Homebrew, which uses different paths). > > While going through the project, I noticed that everything in "protocol" > downloads a pre-built executable, which is simply downloaded and run this > doesn’t seem to run on Arm64 (new M1 and M2 Macs). > > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). > > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture, > so we can use that in
AW: [DISCUSS] Refactor the way we're building/using Thrift
Well it seems it’s not quite that simple. Because for example the CMakeList.txt references directories outside of the module itself. https://github.com/apache/iotdb/blob/master/iotdb-client/client-cpp/src/main/CMakeLists.txt If you wanted to build the client without having built thrift, this will not work. I’m currently working on making the thrift module sort of independent from the rest. The idea is to build the artifact once on every target platform and to make the produced artifacts available via Maven. Instead of just having one executable downloaded, I’d propose to have an archive containing the: * Thrift executable * Thrift runtime shared libraries * Thrift header files With this every module, would just unpack that archive and use that locally. This removes any circular dependencies. However, this module sort of doesn’t fit nicely into the build and it will cause problems. It will always just build thrift for the one platform the RM is currently using. So, I would propose to move this tiny project into the other git repository, that we currently use to serve the pre-compiled binaries. There whenever we need a new OS, CPU Architecture or Thrift version, we update that module and release it (and stage new thrift assembies in Nexus) This would eliminate all complexities in the build. Chris Von: Xiangdong Huang Datum: Samstag, 29. Juli 2023 um 08:37 An: dev@iotdb.apache.org Betreff: Re: [DISCUSS] Refactor the way we're building/using Thrift Hi Chris, > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). I quickly check current compile-tool, now it is only used by client-cpp. So is it really necessary to move to the root folder? > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). Yes, I also agree to compile tools ourselves, rather than download from somewhere.. (and just run once for each operation system.) > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture +1. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Christofer Dutz 于2023年7月28日周五 20:42写道： > > Hi, > > So, I've been fighting to simplify our use of Thrift, as there were several > problems with how we do it today on my M2 Mac (Mostly however to an updated > Homebrew, which uses different paths). > > While going through the project, I noticed that everything in "protocol" > downloads a pre-built executable, which is simply downloaded and run this > doesn’t seem to run on Arm64 (new M1 and M2 Macs). > > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). > > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture, > so we can use that in the client-cpp and the client-cpp-example. > I'd like to give it a try if we could also use this approach for the libs and > completely get rid of the thrift compilation as part of the build. > > What do you think? > > Chris >
Re: [DISCUSS] Refactor the way we're building/using Thrift
Hi Chris, > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). I quickly check current compile-tool, now it is only used by client-cpp. So is it really necessary to move to the root folder? > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). Yes, I also agree to compile tools ourselves, rather than download from somewhere.. (and just run once for each operation system.) > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture +1. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Christofer Dutz 于2023年7月28日周五 20:42写道： > > Hi, > > So, I've been fighting to simplify our use of Thrift, as there were several > problems with how we do it today on my M2 Mac (Mostly however to an updated > Homebrew, which uses different paths). > > While going through the project, I noticed that everything in "protocol" > downloads a pre-built executable, which is simply downloaded and run this > doesn’t seem to run on Arm64 (new M1 and M2 Macs). > > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). > > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture, > so we can use that in the client-cpp and the client-cpp-example. > I'd like to give it a try if we could also use this approach for the libs and > completely get rid of the thrift compilation as part of the build. > > What do you think? > > Chris >
[DISCUSS] Refactor the way we're building/using Thrift
Hi, So, I've been fighting to simplify our use of Thrift, as there were several problems with how we do it today on my M2 Mac (Mostly however to an updated Homebrew, which uses different paths). While going through the project, I noticed that everything in "protocol" downloads a pre-built executable, which is simply downloaded and run this doesn’t seem to run on Arm64 (new M1 and M2 Macs). I would like to move the compile-tools directory into the root of the project and detach it from the rest of the project (So it’s not called from the build if we build the project). This only needs to be run every now and then if something fundamental changes (Like new OSes or new CPU Architectures). I would like to update that module, so it will not only build an executable, but also package the includes and the libraries for that OS and Architecture, so we can use that in the client-cpp and the client-cpp-example. I'd like to give it a try if we could also use this approach for the libs and completely get rid of the thrift compilation as part of the build. What do you think? Chris
