Re: [DISCUSS] Updating to a newer Thrift version?
Hi, Chris In the latest version of thrift, there are some improvements we'd like to make, For example, https://issues.apache.org/jira/browse/THRIFT-5502 will down-cycled "connection reset" warn logs. +1 for upgrading thrift. Thanks Xinyu Tan > 2023年7月31日 19:50，Christofer Dutz 写道： > > Hi all, > > While working on the cleanup of the build, I noticed we’re working with > Thrift in version 0.14.1 however the latest version is 0.18.1 > > Is there a reason we’re sticking to a two-year older version than the newest? > > If not: with the pom-cleanup refactoring it should be a thing of minutes to > update this. > > Chris >
AW: [DISCUSS] Updating to a newer Thrift version?
Ok .. and now I know why we’re at 0.14.1. Turns out 0.17.0 has an issue in its MANIFEST.mf which causes our build to fail. According to Maven-Central do all older versions till 0.14.1 have known vulnerabilities, so we shouldn’t choose any of those. Turns out that 0.14.1 is the sweet spot where we support Java 8, are able to build and don’t inherit any CVEs. Perhaps we should discuss dropping Java 8 support. Chris Von: Christofer Dutz Datum: Montag, 31. Juli 2023 um 16:06 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Updating to a newer Thrift version? So, inspecting all versions between 0.14.1 and 0.18.1 it turns out 0.17.0 was the last one able to be used in Java 8 applications. I therefore updated my branch to that version. Chris Von: Christofer Dutz Datum: Montag, 31. Juli 2023 um 15:53 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Updating to a newer Thrift version? Hi all, well … cancel that … it seems that Thrift 0.18.1 is compiled with newer java versions and we can no longer, consume the jar in the Java 8 build. Chris Von: Christofer Dutz Datum: Montag, 31. Juli 2023 um 15:50 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Updating to a newer Thrift version? Aaaah … ok That’s an explanation … I did try the build in my branch with 0.18.1 and it worked nicely. Chris Von: Yuan Tian Datum: Montag, 31. Juli 2023 um 14:45 An: dev@iotdb.apache.org Betreff: Re: [DISCUSS] Updating to a newer Thrift version? Hi Chris, I remember that we tried to upgrade to 0.15.X before, but the performance declined. I'm not sure about that? Anyone else remember? Best, --- Yuan Tian
AW: [DISCUSS] Updating to a newer Thrift version?
So, inspecting all versions between 0.14.1 and 0.18.1 it turns out 0.17.0 was the last one able to be used in Java 8 applications. I therefore updated my branch to that version. Chris Von: Christofer Dutz Datum: Montag, 31. Juli 2023 um 15:53 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Updating to a newer Thrift version? Hi all, well … cancel that … it seems that Thrift 0.18.1 is compiled with newer java versions and we can no longer, consume the jar in the Java 8 build. Chris Von: Christofer Dutz Datum: Montag, 31. Juli 2023 um 15:50 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Updating to a newer Thrift version? Aaaah … ok That’s an explanation … I did try the build in my branch with 0.18.1 and it worked nicely. Chris Von: Yuan Tian Datum: Montag, 31. Juli 2023 um 14:45 An: dev@iotdb.apache.org Betreff: Re: [DISCUSS] Updating to a newer Thrift version? Hi Chris, I remember that we tried to upgrade to 0.15.X before, but the performance declined. I'm not sure about that? Anyone else remember? Best, --- Yuan Tian
AW: [DISCUSS] Updating to a newer Thrift version?
Hi all, well … cancel that … it seems that Thrift 0.18.1 is compiled with newer java versions and we can no longer, consume the jar in the Java 8 build. Chris Von: Christofer Dutz Datum: Montag, 31. Juli 2023 um 15:50 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Updating to a newer Thrift version? Aaaah … ok That’s an explanation … I did try the build in my branch with 0.18.1 and it worked nicely. Chris Von: Yuan Tian Datum: Montag, 31. Juli 2023 um 14:45 An: dev@iotdb.apache.org Betreff: Re: [DISCUSS] Updating to a newer Thrift version? Hi Chris, I remember that we tried to upgrade to 0.15.X before, but the performance declined. I'm not sure about that? Anyone else remember? Best, --- Yuan Tian
AW: [DISCUSS] Updating to a newer Thrift version?
Aaaah … ok That’s an explanation … I did try the build in my branch with 0.18.1 and it worked nicely. Chris Von: Yuan Tian Datum: Montag, 31. Juli 2023 um 14:45 An: dev@iotdb.apache.org Betreff: Re: [DISCUSS] Updating to a newer Thrift version? Hi Chris, I remember that we tried to upgrade to 0.15.X before, but the performance declined. I'm not sure about that? Anyone else remember? Best, --- Yuan Tian
Re: [DISCUSS] Updating to a newer Thrift version?
Hi Chris, I remember that we tried to upgrade to 0.15.X before, but the performance declined. I'm not sure about that? Anyone else remember? Best, --- Yuan Tian
[DISCUSS] Updating to a newer Thrift version?
Hi all, While working on the cleanup of the build, I noticed we’re working with Thrift in version 0.14.1 however the latest version is 0.18.1 Is there a reason we’re sticking to a two-year older version than the newest? If not: with the pom-cleanup refactoring it should be a thing of minutes to update this. Chris
AW: [DISCUSS] Refactor the way we're building/using Thrift
And again sorry, turns out I selected the branches the wrong way around … this is the right PR: https://github.com/apache/iotdb/pull/10742 Chris Von: Christofer Dutz Datum: Montag, 31. Juli 2023 um 09:33 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift Hi Folks, So, I’ve literally spent all weekend working on my refactoring and must admit I’m quite happy with the results. However, did I decide to take the PR out of this repo and work on a fork in my GitHub account as I didn’t want to swamp you folks with build failure emails. So, I closed the old PR and opened a new one: https://github.com/apache/iotdb/pull/10741 Right now, I still have two TODOs that I would like to work on: * Make the cpp-client build again on Windows (AARCH64 and x86_64) (Currently struggling with it not detecting that it needs a different socket implementation) * Introduce an “iotdb-test-server” module, that integration tests use for running tests against a server: * The current usage of calling scripts in distribution introduces a logical cycle in Maven (Example: cpp-client needs distribution to run tests, distribution bundles cpp-client) * I had many build failures, as the IoTDB server instances are not shut down cleanly. I have noticed that we generally seem to have a number of flaky tests, that randomly fail … as I’ve been running dozens of full builds with integration tests on all supported platforms, I’ve had to re-run many builds to continue from the last failure. I’ve started compiling a list of these tests and I think we should fix them, as the CI should generally pass and not require an arbitrary number of re-runs. Chris Von: Christofer Dutz Datum: Samstag, 29. Juli 2023 um 16:22 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift Well it seems it’s not quite that simple. Because for example the CMakeList.txt references directories outside of the module itself. https://github.com/apache/iotdb/blob/master/iotdb-client/client-cpp/src/main/CMakeLists.txt If you wanted to build the client without having built thrift, this will not work. I’m currently working on making the thrift module sort of independent from the rest. The idea is to build the artifact once on every target platform and to make the produced artifacts available via Maven. Instead of just having one executable downloaded, I’d propose to have an archive containing the: * Thrift executable * Thrift runtime shared libraries * Thrift header files With this every module, would just unpack that archive and use that locally. This removes any circular dependencies. However, this module sort of doesn’t fit nicely into the build and it will cause problems. It will always just build thrift for the one platform the RM is currently using. So, I would propose to move this tiny project into the other git repository, that we currently use to serve the pre-compiled binaries. There whenever we need a new OS, CPU Architecture or Thrift version, we update that module and release it (and stage new thrift assembies in Nexus) This would eliminate all complexities in the build. Chris Von: Xiangdong Huang Datum: Samstag, 29. Juli 2023 um 08:37 An: dev@iotdb.apache.org Betreff: Re: [DISCUSS] Refactor the way we're building/using Thrift Hi Chris, > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). I quickly check current compile-tool, now it is only used by client-cpp. So is it really necessary to move to the root folder? > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). Yes, I also agree to compile tools ourselves, rather than download from somewhere.. (and just run once for each operation system.) > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture +1. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Christofer Dutz 于2023年7月28日周五 20:42写道： > > Hi, > > So, I've been fighting to simplify our use of Thrift, as there were several > problems with how we do it today on my M2 Mac (Mostly however to an updated > Homebrew, which uses different paths). > > While going through the project, I noticed that everything in "protocol" > downloads a pre-built executable, which is simply downloaded and run this > doesn’t seem to run on Arm64 (new M1 and M2 Macs). > > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). > > This only needs to be run every now and then if something fundamental changes > (Like new OSes or
AW: [DISCUSS] Refactor the way we're building/using Thrift
Hi Folks, So, I’ve literally spent all weekend working on my refactoring and must admit I’m quite happy with the results. However, did I decide to take the PR out of this repo and work on a fork in my GitHub account as I didn’t want to swamp you folks with build failure emails. So, I closed the old PR and opened a new one: https://github.com/apache/iotdb/pull/10741 Right now, I still have two TODOs that I would like to work on: * Make the cpp-client build again on Windows (AARCH64 and x86_64) (Currently struggling with it not detecting that it needs a different socket implementation) * Introduce an “iotdb-test-server” module, that integration tests use for running tests against a server: * The current usage of calling scripts in distribution introduces a logical cycle in Maven (Example: cpp-client needs distribution to run tests, distribution bundles cpp-client) * I had many build failures, as the IoTDB server instances are not shut down cleanly. I have noticed that we generally seem to have a number of flaky tests, that randomly fail … as I’ve been running dozens of full builds with integration tests on all supported platforms, I’ve had to re-run many builds to continue from the last failure. I’ve started compiling a list of these tests and I think we should fix them, as the CI should generally pass and not require an arbitrary number of re-runs. Chris Von: Christofer Dutz Datum: Samstag, 29. Juli 2023 um 16:22 An: dev@iotdb.apache.org Betreff: AW: [DISCUSS] Refactor the way we're building/using Thrift Well it seems it’s not quite that simple. Because for example the CMakeList.txt references directories outside of the module itself. https://github.com/apache/iotdb/blob/master/iotdb-client/client-cpp/src/main/CMakeLists.txt If you wanted to build the client without having built thrift, this will not work. I’m currently working on making the thrift module sort of independent from the rest. The idea is to build the artifact once on every target platform and to make the produced artifacts available via Maven. Instead of just having one executable downloaded, I’d propose to have an archive containing the: * Thrift executable * Thrift runtime shared libraries * Thrift header files With this every module, would just unpack that archive and use that locally. This removes any circular dependencies. However, this module sort of doesn’t fit nicely into the build and it will cause problems. It will always just build thrift for the one platform the RM is currently using. So, I would propose to move this tiny project into the other git repository, that we currently use to serve the pre-compiled binaries. There whenever we need a new OS, CPU Architecture or Thrift version, we update that module and release it (and stage new thrift assembies in Nexus) This would eliminate all complexities in the build. Chris Von: Xiangdong Huang Datum: Samstag, 29. Juli 2023 um 08:37 An: dev@iotdb.apache.org Betreff: Re: [DISCUSS] Refactor the way we're building/using Thrift Hi Chris, > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). I quickly check current compile-tool, now it is only used by client-cpp. So is it really necessary to move to the root folder? > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). Yes, I also agree to compile tools ourselves, rather than download from somewhere.. (and just run once for each operation system.) > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture +1. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Christofer Dutz 于2023年7月28日周五 20:42写道： > > Hi, > > So, I've been fighting to simplify our use of Thrift, as there were several > problems with how we do it today on my M2 Mac (Mostly however to an updated > Homebrew, which uses different paths). > > While going through the project, I noticed that everything in "protocol" > downloads a pre-built executable, which is simply downloaded and run this > doesn’t seem to run on Arm64 (new M1 and M2 Macs). > > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). > > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture, > so we can use that in the client-cpp and the client-cpp-example. > I'd like to give it a try if we could also use
