Re: DISCUSS: merge calcite-avatica and calcite repositories
Vladimir, I proposed a straw poll to try to move this thread from a shouting match to a productive conversation. Straw polls are an imperfect mechanism but can sometimes be helpful. It's too bad you didn't find it helpful. I did and I hope others did. > Suppose PR verification passes. > Suppose I merge the PR. > Then "post-merge CI job fails". > Who resolves the issue? > In the proposal, there is no issue with the post-merge CI job failing. It's simply the fyi: "the Avatica upgrade is going to be more than a version change". Everybody can ignore it. > I'm strongly against moving the burden of such maintenance to committers. > The newly added code should not contain known bugs. > There is no burden in this proposal beyond changing the CI. From the Calcite codebase's perspective, Calcite could stay on an old Avatica release forever. This is no different than any other library Calcite sources: if someone wants to upgrade the version of any library that Calcite depends on, they are responsible for solving any incompatibilities before pointing at the new version. The survey is biased, so are the outcomes. > :D On the one hand, all surveys are biased unless there is substantial rigor during construction. On the other hand, I doubt you're referencing the general weakness of simple surveys. I think you're questioning my underlying motivation. In that you're seeing something that isn't there. I have zero opinion on whether to merge or not (and didn't respond to the survey). > Why don't you ask the other questions raised in the thread? > The longer the survey, the less responses one expects to get. The survey was probably already too long given that we only got 7 responses.
Re: DISCUSS: merge calcite-avatica and calcite repositories
Jacques>Based on this, I would suggest we do a small Jacques>modification around CI on Calcite whereby pre-commit tests do not include Jacques>running against Avatica main and instead we have a post-merge CI job that Jacques>monitors the status of connecting the two Suppose PR verification passes. Suppose I merge the PR. Then "post-merge CI job fails". Who resolves the issue? I'm strongly against moving the burden of such maintenance to committers. The newly added code should not contain known bugs. Julian>CI of Calcite master against Avatica master SHOULD pass but not MUST The issue is that people rarely watch for the status of "allowed failures". The PR verification either succeeds or fails. If it succeeds, then you are done. No one would and no one should open "non-mandatory verifications" each and every time. It is like @Disabled tests. Once the test gets ignored, people rarely revisit it, even if they promise to heal it in a couple of weeks (see CALCITE-3457). So the earlier we detect the incompatibility, the easier it is to fix. Jacques>My high level observations The survey is biased, so are the outcomes. There's a question regarding "separate release cycles". How that question contributes to "should we merge ..."? Why don't you ask the other questions raised in the thread? For instance: * How many Avatica versions should Calcite support?" * Have you ever used source release artifacts (*.tar.gz) outside of "release voting"? "never", "Calcite", "Avatica", "other ASF projects" * Would you prefer voting and testing on Calcite and Avatica separately or would you prefer to have a single release candidate? * Is it important to be able to commit Avatica changes while Calcite is being released? * Would you prefer having different release processes for Avatica-java and Calcite, or would you prefer to have a single sequence to release both of them? * Is it important to be able to open both Avatica and Calcite in a single IDE for making changes and running tests? "nice to have", "don't care", "not important" * Is it important to be able to file PRs touching Avatica and Calcite at the same time? "nice to have", "don't care" * Is it important to have the same CI and build scripts among Avatica and Calcite?: "same scripts please", "don't care", "different build scripts (or even build systems) are fine" * Is aligning test framework code between Avatica and Calcite (e.g. junit, hamcrest versions) important? "nice to have", "don't care" * Is it important to have Avatica and Calcite as separate artifacts at dist.apache.org? "nice to have", "don't care" * Is it important to be able to separate issue and PR notifications for Avatica and Calcite? And so on. Vladimir
Re: DISCUSS: merge calcite-avatica and calcite repositories
We've had 7 responses so far on the straw poll. For a reminder, results are here: https://docs.google.com/spreadsheets/d/1EXhrRkYp2owGAK2e3bBw1bEvNj5O3V22iTqndBAuyG8/edit?usp=sharing My high level observations: - There isn't a lot of agreement on most questions :D. - 70% of people preferred to keep the repositories separate (5 out of 7). - All of the people who said they would be willing to "Maintain/enhance/support the Avatica build/CI" also thought the repositories should stay separate. Since maintenance is such a big part of the ongoing cost of projects and the community average slightly prefers to keep things separate, I suggest we stick with separate. Additionally, I really appreciated Julian's comment on the survey, pasted here "CI of Calcite master against Avatica master SHOULD pass but not MUST. It is advisory only.". Based on this, I would suggest we do a small modification around CI on Calcite whereby pre-commit tests do not include running against Avatica main and instead we have a post-merge CI job that monitors the status of connecting the two. This way people have "green" tests for pre-merge even if the latest Avatica doesn't match. I'd also suggest that if Calcite and Avatica are truly treated as decoupled, this allows Calcite to be released not necessarily using the latest Avatica release (treating it's upgrade policy just like any other library Calcite uses). On Fri, Nov 12, 2021 at 8:06 AM Vladimir Sitnikov < sitnikov.vladi...@gmail.com> wrote: > Josh>I am still upset from the porting of Avatica to Gradle > > Please migrate Avatica to whatever build system you like. > > I disagree with you a lot, however, I stop commenting as I do not see it > might change your mind. > I'm not touching Avatica anymore. > > me>I am going to delete/ignore the offending tests in Calcite in 24 hours > > Done. I commented out the test. > I hope we no longer add tests to Caclite that verify low-level Avatica > behaviour. > > Vladimir >
Re: DISCUSS: merge calcite-avatica and calcite repositories
Josh>I am still upset from the porting of Avatica to Gradle Please migrate Avatica to whatever build system you like. I disagree with you a lot, however, I stop commenting as I do not see it might change your mind. I'm not touching Avatica anymore. me>I am going to delete/ignore the offending tests in Calcite in 24 hours Done. I commented out the test. I hope we no longer add tests to Caclite that verify low-level Avatica behaviour. Vladimir
Re: DISCUSS: merge calcite-avatica and calcite repositories
Java 17 is not a priority to me. And, there were few changes in Avatica over the time range in question, so there is no benefit to having releases "for free". Most likely, having additional releases for Avatica would have created _more_ work for me downstream through my employer and Phoenix's integration of Avatica. This is all coming back to my opinion that Avatica and Calcite, though governed by the same organization, do not have to have their development done in lock-step. Yes, there can be benefit to combining build metadata and configuration, but I do not believe the number of developers nor the automated builds for Avatica require such consolidation (Avatica is just not that complicated). I am still upset from the porting of Avatica to Gradle and the amount of effort which I came into because of the opinion that "Gradle is better". I burned at least one week just to get Avatica "back" to a state where I could actually do development on it and downstream integration was again working. It's extremely hard for me to look at your suggestions to move the code base and change the build configuration, and not feel like I'm going to end up having even more debt to pay down. I should apologize for my heated remarks earlier; I was quite upset that my original comment about combining avatica and avatica-go was taken as I was OK with combining calcite and avatica. tl;dr Avatica is not a high-activity project and that is OK. I do not want to worry about a shared build configuration between Calcite and Avatica because I want to be able to make changes to Avatica _that make sense for Avatica_. I don't have nearly as much time as I wish I had to write new code; this is why I am not in favor of such fundamental change without a very good reason (acknowledging: this is where you and I disagree, there are large problems and the changes you are suggesting would fix those). Thanks for putting the poll out, Jacques. On 11/10/21 4:33 PM, Vladimir Sitnikov wrote: Josh>because I genuinely do not see how this approach Josh>creates more code that needs to be written than if the projects were in Josh>the same repository. Compare https://github.com/apache/calcite-avatica/blob/4cf769f8ee32bb3520604e4f3284e6f233f90276/build.gradle.kts and https://github.com/apache/calcite/blob/6d51d277b158ff7073f29ada4a96a7a74c0b46fc/build.gradle.kts Those are duplicate files, and I had to create them twice. If the code was co-located in a single repository, then a single copy of the file would be enough. John>(I am to assume) is a personal motivation for yourself. Everybody who contributes to the build scripts, dependencies, etc, would want to avoid duplication of their work. John>Avatica 1.17.0 was released Jun 2020, 1.18.0 in May 2021. Calcite had 3 John>releases of its own between these. In other words, Avatica could have three releases **for free** Of course, you might argue there's "non-trivial procedure for releasing Calcite", however, I would claim it does not have to be that way. "lots of manual steps for the release" add zero to the quality of the release. Of course, we might disagree here as well, however, if releasing Calcite is painful, then "forking Avatica into a different repository" is a wrong solution. John>I can still build Avatica with Gradle 6.8.1. Go ahead and try building Avatica with Java 17. You will fail quite fast. Have a fun day. Vladimir
Re: DISCUSS: merge calcite-avatica and calcite repositories
I'm finding it difficult to make sense of all the responses on this thread. I put up a quick straw poll that will hopefully try to give us a better sense of where people's heads are at. I think the thread outlines a number of concerns for both outcomes. I'd ask people to read the thread and then share their feedback on the survey. Survey: https://forms.gle/hUZaEP3aqcbe31HC9 Results: https://docs.google.com/spreadsheets/d/1EXhrRkYp2owGAK2e3bBw1bEvNj5O3V22iTqndBAuyG8/edit?usp=sharing Hopefully this will cut through some of the noise. On Wed, Nov 10, 2021 at 1:35 PM Vladimir Sitnikov < sitnikov.vladi...@gmail.com> wrote: > s/John/Josh/ > > sorry for the typo. > > Vladimir >
Re: DISCUSS: merge calcite-avatica and calcite repositories
s/John/Josh/ sorry for the typo. Vladimir
Re: DISCUSS: merge calcite-avatica and calcite repositories
Josh>because I genuinely do not see how this approach Josh>creates more code that needs to be written than if the projects were in Josh>the same repository. Compare https://github.com/apache/calcite-avatica/blob/4cf769f8ee32bb3520604e4f3284e6f233f90276/build.gradle.kts and https://github.com/apache/calcite/blob/6d51d277b158ff7073f29ada4a96a7a74c0b46fc/build.gradle.kts Those are duplicate files, and I had to create them twice. If the code was co-located in a single repository, then a single copy of the file would be enough. John>(I am to assume) is a personal motivation for yourself. Everybody who contributes to the build scripts, dependencies, etc, would want to avoid duplication of their work. John>Avatica 1.17.0 was released Jun 2020, 1.18.0 in May 2021. Calcite had 3 John>releases of its own between these. In other words, Avatica could have three releases **for free** Of course, you might argue there's "non-trivial procedure for releasing Calcite", however, I would claim it does not have to be that way. "lots of manual steps for the release" add zero to the quality of the release. Of course, we might disagree here as well, however, if releasing Calcite is painful, then "forking Avatica into a different repository" is a wrong solution. John>I can still build Avatica with Gradle 6.8.1. Go ahead and try building Avatica with Java 17. You will fail quite fast. Have a fun day. Vladimir
Re: DISCUSS: merge calcite-avatica and calcite repositories
On 11/10/21 3:48 PM, Vladimir Sitnikov wrote: Josh>If we identify these problems, we can think Josh>through (from the beginning) if combining repositories is the only Josh>solution or just one possible solution (and weigh the pros and cons for Josh>each). I am sure I did explain the pain points. See points 1..5 (the very beginning of the mail) in the very first message in this thread. If that is not enough, I can't really explain better. CI configuration is duplicated across repositories. JUnit versions maintenance overheads are duplicated. Build configuration is duplicated (e.g. code style rules, boilerplate build scripts, etc) Cross-repository PRs are non-trivial. And so on. These are not things that are broken. CI configuration, JUnit versions, and build configuration are already defined for each project. If you want to change them, you can submit a change and test them. It sounds like you want to be able to change them in exactly one place which (I am to assume) is a personal motivation for yourself. Cross-repository PRs, I've already addressed that this is a solved problem. Please illustrate an example how this creates more code that you have to write, because I genuinely do not see how this approach creates more code that needs to be written than if the projects were in the same repository. 1. Update avatica 2. Release avatica 3. Upgrade avatica dependency in calcite with corresponding changes due to avatica upgrade. In your example where avatica is in the code base, it discourages API semantics to *not* be compatible by allowing you to quickly change the Avatica API and immediately reflect those changes in Calcite. If folks start doing this, then you disrupt the entire Calcite developer base who has to realize that the API they wanted to use is no longer the correct API. > If you have suggestions on how to bump JUnit and CI configuration in both repositories at once You are correct in that I do not believe this to be a "problem". > It is clear that having the repositories separate has no real benefits. This is not a fact, but this is your opinion. > committers are the same The PMC is the same and those authorized to make commits to each are the same group, but the individuals are *not*. For example, I know damn well that I should not go merging changes that I make into Calcite because I do not regularly do this. In the same way, I'd expect 90%+ of the Calcite committers have never contributed to Avatica. If your opinion is some philosophical issue with the PMC governance being shared between Avatica and Calcite, we can have that discussion instead. > I am sure those "reasons" have no weight Again, this is your opinion. The fact is that I have made Avatica releases without the burden of making a Calcite release, and I know that multiple Calcite releases have been made without the burden of making a new Avatica release. This is not "theoretical". Avatica 1.17.0 was released Jun 2020, 1.18.0 in May 2021. Calcite had 3 releases of its own between these. > If so, I just let them do the maintenance. That is what I would prefer you to do. > Avatica is stuck with Gradle 6.8.1 I can still build Avatica with Gradle 6.8.1. Let me be clear to remove all doubt at this point in time: I am -1 on combining these repositories based on the current discussion.
Re: DISCUSS: merge calcite-avatica and calcite repositories
My mistake. I originally wrote “If we want to keep our linear commit history
(i.e. avoid a huge merge commit), all of the commit hashes will have to
change”. I believe that is a true statement. I shouldn’t have tried to simplify
it.
> On Nov 10, 2021, at 3:46 AM, Dmitry Sysolyatin
> wrote:
>
> Actually, it is possible to preserve the history of commits when you merge
> two repos -
> https://stackoverflow.com/questions/17371150/moving-git-repository-content-to-another-repository-preserving-history
>
> On Tue, Nov 9, 2021 at 11:24 PM Julian Hyde wrote:
>
>> Merging the repositories is unnecessary, and a massive waste of time.
>>
>> Any supposed “time savings” due to the the merged repositories will be
>> eaten up by dozens of small issues that will crop up long after the merge.
>>
>> I was release manager for Calcite and Avatica this time around and it took
>> me about a week, because the process had changed since last time I was
>> release manager. And now we’re going to change the process again?
>> Absolutely fucking insane.
>>
>> Also, we will lose the history of the Avatica source code. All of the
>> commit hashes for bugs fixed long ago will change.
>>
>> At the time that we split the repositories, I was probably against the
>> idea, because it was work. Now that we’ve split, I am strongly against
>> recombining the repositories.
>>
>> Honestly, people, we have better things to do.
>>
>> Julian
>>
>>
>>> On Nov 9, 2021, at 1:03 PM, Stamatis Zampetakis
>> wrote:
>>>
>>> The two projects have more or less the same release cadence, they're
>>> maintained by the same community, they're written in the same language,
>> and
>>> they use the same build system.
>>>
>>> Although I understand the reasons the project was split in the first
>> place,
>>> at the moment I tend to agree that having them in the same repo would be
>>> more practical.
>>>
>>> Truth is the merge may require quite a bit of work so we should take this
>>> into consideration.
>>>
>>> If people prefer keeping the projects as they are I am perfectly fine
>> with
>>> that as well. I am leaving the decision to those who feel strongly about
>>> this change.
>>>
>>> Best,
>>> Stamatis
>>>
>>>
>>>
>>> On Tue, Nov 9, 2021, 7:46 PM Vladimir Ozerov wrote:
>>>
+1 for a single repo.
Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov <
sitnikov.vladi...@gmail.com
> :
> Michael> is not proposing to change the
> Michael>structure of modules within both projects, merely to have the
code
> for
> Michael>both in a single repository.
>
> I propose to integrate them into a single build, and keep the set of
>> the
> published jars.
> However, the modules and dependency structure could be kept as is.
>
> We might want to rename folders like
> calcite-core
> calcite-linq4j
> ..
> avatica-core
> avatica-server
>
> However, I am not sure it is that important to discuss folder names
>> now.
> The idea is that as you "open Calcite in IDE, you see both Avatica and
> Calcite modules"
>
> Michael>Is there any reason we couldn't have a separate release
>> schedule
if
> Michael>both projects are in the same repository?
>
> A different schedule means Calcite must support at least two different
> Avatica versions.
> In other words, if we allow clients to update Avatica at their will,
>> then
> we should allow them building Calcite with different Avatica versions,
> which implies Calcite test code should succeed for multiple different
> Avatica vesions.
>
> That makes it harder to write tests: we have to execute tests with two
> different Avatica releases (or even more than two).
>
> There are at least two sources for complexity:
>
> a) We have to write tests that tolerate multiple versions. For
>> instance,
> "if (avatica.18+) {...}" and so on.
> That is not really trivial, especially taking into account some of the
> tests are created with non-yet-popular
> technologies like Quidem where you can't really google solutions. So
>> the
> "trivial" task of "making a test to expect two possible outcomes"
> becomes less trivial as you try to pass the version from GitHub Action
>> to
> Gradle to JUnit to Quidem to no-one-knows-which class.
> If we support one Avatica version only, that is not needed. We just
>> patch
> the test in Avatica and Calcite and that is it.
> Single repo avoids "Gradle vs Quidem" dance.
>
> b) If we claim that we support 5 different Guava versions, 3 different
JDK
> versions, 2 different Avatica versions,
> then we have to execute 5*3*2 = 30 combinations of the tests.
> That is not really a full matrix, however, things get way easier if we
> support one Avatica version only.
> The amount of tests we need to do during a proper release is much less,
and
> it is
Re: DISCUSS: merge calcite-avatica and calcite repositories
On 11/10/21 3:48 PM, Vladimir Sitnikov wrote: Josh>I have no objections to combining these two repositories together. Why don't we just merge the repositories and move on then? This was in reference to merging calcite-avatica and calcite-avatica-go together. I thought this would be apparent given the rest of my arguments *against* merging them. Sorry I was not in more clear disagreement.
Re: DISCUSS: merge calcite-avatica and calcite repositories
Josh>I have no objections to combining these two repositories together. Why don't we just merge the repositories and move on then? Josh>If we identify these problems, we can think Josh>through (from the beginning) if combining repositories is the only Josh>solution or just one possible solution (and weigh the pros and cons for Josh>each). I am sure I did explain the pain points. See points 1..5 (the very beginning of the mail) in the very first message in this thread. If that is not enough, I can't really explain better. CI configuration is duplicated across repositories. JUnit versions maintenance overheads are duplicated. Build configuration is duplicated (e.g. code style rules, boilerplate build scripts, etc) Cross-repository PRs are non-trivial. And so on. If you have suggestions on how to bump JUnit and CI configuration in both repositories at once, you are welcome, however, the only alternative "solution" I expect is to ignore the problem and treat it as "not a problem". I acknowledge merging repositories will make some use cases harder. However, I am sure points 1..5 (see the very first mail) have no solution besides repository merge. It is clear that having the repositories separate has no real benefits. Community is the same, committers are the same, language is the same, releases are co-located. There are theoretical reasons like "different repositories allow different committers in theory" or "different repositories allow different release schedules" or "because people perceive that they can use component A without using component B" However, I am sure those "reasons" have no weight. They are theoretical, and some of them turned out to be false. For instance, the release schedule is co-located across several years. Even at the time of the Avatica split, somebody said that Solr community did **merge** repositories after a while exactly because the committers were effectively the same. Of course, somebody might **believe** that "because people perceive that they can use component A without using component B" is way more important than a non-important duplication of maintenance of CI, dependencies, build scripts across both repositories. You know what? I am fine to accept people might have those beliefs. If so, I just let them do the maintenance. Of course, you might claim that your ability to watch "avatica PRs only" is far more important. Is that really the key point of having separate repositories? Really? GitHub has "code owners" and "labeling" features, so you can watch the relevant parts of the repository. Josh>These repositories are separate because they have separate release Josh>schedules The release dates for Calcite and Avatica are co-located (see my mail above). The feature of "separate release schedules" is not really used in practice. Having the same version number for Calcite and Avatica would event make it easier for clients to pick the compatible versions. Josh> The net amount of code you have to write doesn't Josh> change with how it works now compared to how you are suggesting it Josh> should work. This is false. For instance, "CI configuration, JUnit versions, build scripts" are **duplicated** across repositories, so if the repositories are different, then the amount of work is increased. Here's one more data point: Calcite is built with Gradle 7.3, and Avatica is stuck with Gradle 6.8.1 Of course, the duplication of work does not exist. You say "the net amount of code I have to write doesn't change". You are right. I won't contribute to calcite-avatica repository anymore, so there's no duplication of work for me. Josh>If there are breaking changes in Avatica, they would need to be Josh>accounted for when Calcite is bumped to the next version of Avatica. calcite-avatica/pull/161 would be way easier if both avatica and calcite were in a single repository. There would be no need to think of version compatibility, etc. Josh> Having a separate repository makes it Josh>extremely easy for me to watch Avatica commits/pull-requests which I am Josh>capable of reviewing vs. Calcite pull-requests which I am not Josh>comfortable reviewing. Have you considered mail filters and/or code owners https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners ? There's an option to automatically label PRs according to the modified files via GitHub Action. Josh>rather than starting from the root problem "How can we Josh>make co-dependent changes easier?" "co-dependent" changes are not the only problem. It was just a trigger. Vladimir
Re: DISCUSS: merge calcite-avatica and calcite repositories
Are you still advocating this change, Vladimir? If this is something you want to continue discussing (I'm not sure if your comment about "[I won't contribute to Avatica]" is saying that you don't want to discuss this anymore), could we try to refocus this on some specific technical problems that come up due to Avatica and Calcite being separate repositories. If we identify these problems, we can think through (from the beginning) if combining repositories is the only solution or just one possible solution (and weigh the pros and cons for each). It seems like you also have some pain points with Calcite that you want to improve, but these are separate from the discussion at hand. Let's try to keep those out of a discussion about combining these two repositories, please. On 11/10/21 6:59 AM, Vladimir Sitnikov wrote: Ok, I did not care when Avatica was split into its own repository, and I assumed Avatica team would maintain the code. That turned out to be not quite true, and it turns out there are non-trivial overheads coming from the split. Of course, everybody can disagree, so if you disagree, you don't need to explain that. We just disagree and that is it. Now I do not care how Avatica would evolve further. It might be even good because Vladimir S won't bother Avatica with mails, PRs, and other random items. What I care much more about is that Calcite tests are broken, the CI reports failure for every commit, and that is really insane. Calcite CI must not fail on every build. I am going to delete/ignore the offending tests in Calcite in 24 hours, and I let the others deal with the case (fix tests, fix Quidem, merge Avatica, or whatever). I would no longer review calcite-avatica PRs. Julian>I was release manager for Calcite and Avatica this time around and it took me about a week Julian, as the repositories are merged, there will be a **single** release process that releases both Calcite and Avatica in a single go. It would reduce the friction, and it would reduce cases where the release procedures and steps are "slightly different". It looks like you assume the release steps would never change, however, in practice, the drift of Java, Docker, OS, and so on versions would affect the release steps anyway. That is why having Avatica and Calcite in a single repository would simplify the release procedure. Julian> And now we’re going to change the process again? I propose to *drop* the release process for "calcite-avatica-java". The release of "calcite" would release both things (in a single calcite-x.y.z.tar.gz file), so the release steps for Calcite would be intact. Technically, that changes the process. In practice, I find that you overreact. The merge would make the release easier. On top of that, it was you who voted for splitting Calcite and Avatica, so it was your conscious decision to accept that "release management would get harder after the split" Julian>Any supposed “time savings” due to the merged repositories will be eaten up by dozens of small issues that will crop up long after the merge The words are not justified. There will always be some random small issues, even in the case we keep repositories separate. On the other hand, there are true issues when repositories are different: * We have two different release processes, so RM have to know the differences * we have slightly different dependency versions, so things like "bump junit" have to be duplicated * CI configuration is duplicated across the repositories, and it does have non-zero maintenance overhead * cross-repository refactorings are harder to do (like the fix for the exception message) Julian, I did put four issues that are real, and they induce noticeable maintenance churn. Java and JUnit will keep updating and you can't prevent that. Please highlight the problems exactly and please stop saying "time savings will be eaten by dozens". That is not a healthy discussion. Also, we will lose the history of the Avatica source code. All of the commit hashes for bugs fixed long ago will change We can make calcite-avatica repository read-only, so all commits will be there. Vladimir
Re: DISCUSS: merge calcite-avatica and calcite repositories
My mistake. I was unaware of the --allow-unrelated-histories flag :) -- Michael Mior mm...@apache.org Le mer. 10 nov. 2021 à 07:34, Dmitry Sysolyatin a écrit : > > No. hashes shouldn't change. > You can try the following steps and check: > > git pull https://github.com/apache/calcite.git > cd calcite > git remote add avatica https://github.com/apache/calcite-avatica > git fetch avatica > git merge avatica/master --allow-unrelated-histories > git add . && git commit -m "test" > > If you check the history log you can see that hash for the commit > https://github.com/apache/calcite-avatica/commit/4cf769f8ee32bb3520604e4f3284e6f233f90276 > remains the same > > On Wed, Nov 10, 2021 at 2:20 PM Michael Mior wrote: > > > > Actually, it is possible to preserve the history of commits when you > > merge > > > two repos - > > > > You can keep the history, but the hashes will necessarily change > > because they are based on a different parent. So it will be possible > > to see the changes, but the hashes would reference the old Avatica > > repository which would no longer be active. > > > > -- > > Michael Mior > > mm...@apache.org > > > > Le mer. 10 nov. 2021 à 06:47, Dmitry Sysolyatin > > a écrit : > > > > > > Actually, it is possible to preserve the history of commits when you > > merge > > > two repos - > > > > > https://stackoverflow.com/questions/17371150/moving-git-repository-content-to-another-repository-preserving-history > > > > > > On Tue, Nov 9, 2021 at 11:24 PM Julian Hyde > > wrote: > > > > > > > Merging the repositories is unnecessary, and a massive waste of time. > > > > > > > > Any supposed “time savings” due to the the merged repositories will be > > > > eaten up by dozens of small issues that will crop up long after the > > merge. > > > > > > > > I was release manager for Calcite and Avatica this time around and it > > took > > > > me about a week, because the process had changed since last time I was > > > > release manager. And now we’re going to change the process again? > > > > Absolutely fucking insane. > > > > > > > > Also, we will lose the history of the Avatica source code. All of the > > > > commit hashes for bugs fixed long ago will change. > > > > > > > > At the time that we split the repositories, I was probably against the > > > > idea, because it was work. Now that we’ve split, I am strongly against > > > > recombining the repositories. > > > > > > > > Honestly, people, we have better things to do. > > > > > > > > Julian > > > > > > > > > > > > > On Nov 9, 2021, at 1:03 PM, Stamatis Zampetakis > > > > wrote: > > > > > > > > > > The two projects have more or less the same release cadence, they're > > > > > maintained by the same community, they're written in the same > > language, > > > > and > > > > > they use the same build system. > > > > > > > > > > Although I understand the reasons the project was split in the first > > > > place, > > > > > at the moment I tend to agree that having them in the same repo > > would be > > > > > more practical. > > > > > > > > > > Truth is the merge may require quite a bit of work so we should take > > this > > > > > into consideration. > > > > > > > > > > If people prefer keeping the projects as they are I am perfectly fine > > > > with > > > > > that as well. I am leaving the decision to those who feel strongly > > about > > > > > this change. > > > > > > > > > > Best, > > > > > Stamatis > > > > > > > > > > > > > > > > > > > > On Tue, Nov 9, 2021, 7:46 PM Vladimir Ozerov > > wrote: > > > > > > > > > >> +1 for a single repo. > > > > >> > > > > >> Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov < > > > > >> sitnikov.vladi...@gmail.com > > > > >>> : > > > > >> > > > > >>> Michael> is not proposing to change the > > > > >>> Michael>structure of modules within both projects, merely to have > > the > > > > >> code > > > > >>> for > > > > >>> Michael>both in a single repository. > > > > >>> > > > > >>> I propose to integrate them into a single build, and keep the set > > of > > > > the > > > > >>> published jars. > > > > >>> However, the modules and dependency structure could be kept as is. > > > > >>> > > > > >>> We might want to rename folders like > > > > >>> calcite-core > > > > >>> calcite-linq4j > > > > >>> .. > > > > >>> avatica-core > > > > >>> avatica-server > > > > >>> > > > > >>> However, I am not sure it is that important to discuss folder names > > > > now. > > > > >>> The idea is that as you "open Calcite in IDE, you see both Avatica > > and > > > > >>> Calcite modules" > > > > >>> > > > > >>> Michael>Is there any reason we couldn't have a separate release > > > > schedule > > > > >> if > > > > >>> Michael>both projects are in the same repository? > > > > >>> > > > > >>> A different schedule means Calcite must support at least two > > different > > > > >>> Avatica versions. > > > > >>> In other words, if we allow clients to update Avatica at their > > will, > > > > then > > > > >>> we should allow them building Calcite with different
Re: DISCUSS: merge calcite-avatica and calcite repositories
No. hashes shouldn't change. You can try the following steps and check: git pull https://github.com/apache/calcite.git cd calcite git remote add avatica https://github.com/apache/calcite-avatica git fetch avatica git merge avatica/master --allow-unrelated-histories git add . && git commit -m "test" If you check the history log you can see that hash for the commit https://github.com/apache/calcite-avatica/commit/4cf769f8ee32bb3520604e4f3284e6f233f90276 remains the same On Wed, Nov 10, 2021 at 2:20 PM Michael Mior wrote: > > Actually, it is possible to preserve the history of commits when you > merge > > two repos - > > You can keep the history, but the hashes will necessarily change > because they are based on a different parent. So it will be possible > to see the changes, but the hashes would reference the old Avatica > repository which would no longer be active. > > -- > Michael Mior > mm...@apache.org > > Le mer. 10 nov. 2021 à 06:47, Dmitry Sysolyatin > a écrit : > > > > Actually, it is possible to preserve the history of commits when you > merge > > two repos - > > > https://stackoverflow.com/questions/17371150/moving-git-repository-content-to-another-repository-preserving-history > > > > On Tue, Nov 9, 2021 at 11:24 PM Julian Hyde > wrote: > > > > > Merging the repositories is unnecessary, and a massive waste of time. > > > > > > Any supposed “time savings” due to the the merged repositories will be > > > eaten up by dozens of small issues that will crop up long after the > merge. > > > > > > I was release manager for Calcite and Avatica this time around and it > took > > > me about a week, because the process had changed since last time I was > > > release manager. And now we’re going to change the process again? > > > Absolutely fucking insane. > > > > > > Also, we will lose the history of the Avatica source code. All of the > > > commit hashes for bugs fixed long ago will change. > > > > > > At the time that we split the repositories, I was probably against the > > > idea, because it was work. Now that we’ve split, I am strongly against > > > recombining the repositories. > > > > > > Honestly, people, we have better things to do. > > > > > > Julian > > > > > > > > > > On Nov 9, 2021, at 1:03 PM, Stamatis Zampetakis > > > wrote: > > > > > > > > The two projects have more or less the same release cadence, they're > > > > maintained by the same community, they're written in the same > language, > > > and > > > > they use the same build system. > > > > > > > > Although I understand the reasons the project was split in the first > > > place, > > > > at the moment I tend to agree that having them in the same repo > would be > > > > more practical. > > > > > > > > Truth is the merge may require quite a bit of work so we should take > this > > > > into consideration. > > > > > > > > If people prefer keeping the projects as they are I am perfectly fine > > > with > > > > that as well. I am leaving the decision to those who feel strongly > about > > > > this change. > > > > > > > > Best, > > > > Stamatis > > > > > > > > > > > > > > > > On Tue, Nov 9, 2021, 7:46 PM Vladimir Ozerov > wrote: > > > > > > > >> +1 for a single repo. > > > >> > > > >> Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov < > > > >> sitnikov.vladi...@gmail.com > > > >>> : > > > >> > > > >>> Michael> is not proposing to change the > > > >>> Michael>structure of modules within both projects, merely to have > the > > > >> code > > > >>> for > > > >>> Michael>both in a single repository. > > > >>> > > > >>> I propose to integrate them into a single build, and keep the set > of > > > the > > > >>> published jars. > > > >>> However, the modules and dependency structure could be kept as is. > > > >>> > > > >>> We might want to rename folders like > > > >>> calcite-core > > > >>> calcite-linq4j > > > >>> .. > > > >>> avatica-core > > > >>> avatica-server > > > >>> > > > >>> However, I am not sure it is that important to discuss folder names > > > now. > > > >>> The idea is that as you "open Calcite in IDE, you see both Avatica > and > > > >>> Calcite modules" > > > >>> > > > >>> Michael>Is there any reason we couldn't have a separate release > > > schedule > > > >> if > > > >>> Michael>both projects are in the same repository? > > > >>> > > > >>> A different schedule means Calcite must support at least two > different > > > >>> Avatica versions. > > > >>> In other words, if we allow clients to update Avatica at their > will, > > > then > > > >>> we should allow them building Calcite with different Avatica > versions, > > > >>> which implies Calcite test code should succeed for multiple > different > > > >>> Avatica vesions. > > > >>> > > > >>> That makes it harder to write tests: we have to execute tests with > two > > > >>> different Avatica releases (or even more than two). > > > >>> > > > >>> There are at least two sources for complexity: > > > >>> > > > >>> a) We have to write tests that tolerate multiple versions. For > > >
Re: DISCUSS: merge calcite-avatica and calcite repositories
> Actually, it is possible to preserve the history of commits when you merge
> two repos -
You can keep the history, but the hashes will necessarily change
because they are based on a different parent. So it will be possible
to see the changes, but the hashes would reference the old Avatica
repository which would no longer be active.
--
Michael Mior
mm...@apache.org
Le mer. 10 nov. 2021 à 06:47, Dmitry Sysolyatin
a écrit :
>
> Actually, it is possible to preserve the history of commits when you merge
> two repos -
> https://stackoverflow.com/questions/17371150/moving-git-repository-content-to-another-repository-preserving-history
>
> On Tue, Nov 9, 2021 at 11:24 PM Julian Hyde wrote:
>
> > Merging the repositories is unnecessary, and a massive waste of time.
> >
> > Any supposed “time savings” due to the the merged repositories will be
> > eaten up by dozens of small issues that will crop up long after the merge.
> >
> > I was release manager for Calcite and Avatica this time around and it took
> > me about a week, because the process had changed since last time I was
> > release manager. And now we’re going to change the process again?
> > Absolutely fucking insane.
> >
> > Also, we will lose the history of the Avatica source code. All of the
> > commit hashes for bugs fixed long ago will change.
> >
> > At the time that we split the repositories, I was probably against the
> > idea, because it was work. Now that we’ve split, I am strongly against
> > recombining the repositories.
> >
> > Honestly, people, we have better things to do.
> >
> > Julian
> >
> >
> > > On Nov 9, 2021, at 1:03 PM, Stamatis Zampetakis
> > wrote:
> > >
> > > The two projects have more or less the same release cadence, they're
> > > maintained by the same community, they're written in the same language,
> > and
> > > they use the same build system.
> > >
> > > Although I understand the reasons the project was split in the first
> > place,
> > > at the moment I tend to agree that having them in the same repo would be
> > > more practical.
> > >
> > > Truth is the merge may require quite a bit of work so we should take this
> > > into consideration.
> > >
> > > If people prefer keeping the projects as they are I am perfectly fine
> > with
> > > that as well. I am leaving the decision to those who feel strongly about
> > > this change.
> > >
> > > Best,
> > > Stamatis
> > >
> > >
> > >
> > > On Tue, Nov 9, 2021, 7:46 PM Vladimir Ozerov wrote:
> > >
> > >> +1 for a single repo.
> > >>
> > >> Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov <
> > >> sitnikov.vladi...@gmail.com
> > >>> :
> > >>
> > >>> Michael> is not proposing to change the
> > >>> Michael>structure of modules within both projects, merely to have the
> > >> code
> > >>> for
> > >>> Michael>both in a single repository.
> > >>>
> > >>> I propose to integrate them into a single build, and keep the set of
> > the
> > >>> published jars.
> > >>> However, the modules and dependency structure could be kept as is.
> > >>>
> > >>> We might want to rename folders like
> > >>> calcite-core
> > >>> calcite-linq4j
> > >>> ..
> > >>> avatica-core
> > >>> avatica-server
> > >>>
> > >>> However, I am not sure it is that important to discuss folder names
> > now.
> > >>> The idea is that as you "open Calcite in IDE, you see both Avatica and
> > >>> Calcite modules"
> > >>>
> > >>> Michael>Is there any reason we couldn't have a separate release
> > schedule
> > >> if
> > >>> Michael>both projects are in the same repository?
> > >>>
> > >>> A different schedule means Calcite must support at least two different
> > >>> Avatica versions.
> > >>> In other words, if we allow clients to update Avatica at their will,
> > then
> > >>> we should allow them building Calcite with different Avatica versions,
> > >>> which implies Calcite test code should succeed for multiple different
> > >>> Avatica vesions.
> > >>>
> > >>> That makes it harder to write tests: we have to execute tests with two
> > >>> different Avatica releases (or even more than two).
> > >>>
> > >>> There are at least two sources for complexity:
> > >>>
> > >>> a) We have to write tests that tolerate multiple versions. For
> > instance,
> > >>> "if (avatica.18+) {...}" and so on.
> > >>> That is not really trivial, especially taking into account some of the
> > >>> tests are created with non-yet-popular
> > >>> technologies like Quidem where you can't really google solutions. So
> > the
> > >>> "trivial" task of "making a test to expect two possible outcomes"
> > >>> becomes less trivial as you try to pass the version from GitHub Action
> > to
> > >>> Gradle to JUnit to Quidem to no-one-knows-which class.
> > >>> If we support one Avatica version only, that is not needed. We just
> > patch
> > >>> the test in Avatica and Calcite and that is it.
> > >>> Single repo avoids "Gradle vs Quidem" dance.
> > >>>
> > >>> b) If we claim that we support 5 different Guava versions, 3 different
> > >> JDK
> > >>> versions, 2 different
Re: DISCUSS: merge calcite-avatica and calcite repositories
Ok, I did not care when Avatica was split into its own repository, and I assumed Avatica team would maintain the code. That turned out to be not quite true, and it turns out there are non-trivial overheads coming from the split. Of course, everybody can disagree, so if you disagree, you don't need to explain that. We just disagree and that is it. Now I do not care how Avatica would evolve further. It might be even good because Vladimir S won't bother Avatica with mails, PRs, and other random items. What I care much more about is that Calcite tests are broken, the CI reports failure for every commit, and that is really insane. Calcite CI must not fail on every build. I am going to delete/ignore the offending tests in Calcite in 24 hours, and I let the others deal with the case (fix tests, fix Quidem, merge Avatica, or whatever). I would no longer review calcite-avatica PRs. Julian>I was release manager for Calcite and Avatica this time around and it took me about a week Julian, as the repositories are merged, there will be a **single** release process that releases both Calcite and Avatica in a single go. It would reduce the friction, and it would reduce cases where the release procedures and steps are "slightly different". It looks like you assume the release steps would never change, however, in practice, the drift of Java, Docker, OS, and so on versions would affect the release steps anyway. That is why having Avatica and Calcite in a single repository would simplify the release procedure. Julian> And now we’re going to change the process again? I propose to *drop* the release process for "calcite-avatica-java". The release of "calcite" would release both things (in a single calcite-x.y.z.tar.gz file), so the release steps for Calcite would be intact. Technically, that changes the process. In practice, I find that you overreact. The merge would make the release easier. On top of that, it was you who voted for splitting Calcite and Avatica, so it was your conscious decision to accept that "release management would get harder after the split" Julian>Any supposed “time savings” due to the merged repositories will be eaten up by dozens of small issues that will crop up long after the merge The words are not justified. There will always be some random small issues, even in the case we keep repositories separate. On the other hand, there are true issues when repositories are different: * We have two different release processes, so RM have to know the differences * we have slightly different dependency versions, so things like "bump junit" have to be duplicated * CI configuration is duplicated across the repositories, and it does have non-zero maintenance overhead * cross-repository refactorings are harder to do (like the fix for the exception message) Julian, I did put four issues that are real, and they induce noticeable maintenance churn. Java and JUnit will keep updating and you can't prevent that. Please highlight the problems exactly and please stop saying "time savings will be eaten by dozens". That is not a healthy discussion. >Also, we will lose the history of the Avatica source code. All of the commit hashes for bugs fixed long ago will change We can make calcite-avatica repository read-only, so all commits will be there. Vladimir
Re: DISCUSS: merge calcite-avatica and calcite repositories
Actually, it is possible to preserve the history of commits when you merge
two repos -
https://stackoverflow.com/questions/17371150/moving-git-repository-content-to-another-repository-preserving-history
On Tue, Nov 9, 2021 at 11:24 PM Julian Hyde wrote:
> Merging the repositories is unnecessary, and a massive waste of time.
>
> Any supposed “time savings” due to the the merged repositories will be
> eaten up by dozens of small issues that will crop up long after the merge.
>
> I was release manager for Calcite and Avatica this time around and it took
> me about a week, because the process had changed since last time I was
> release manager. And now we’re going to change the process again?
> Absolutely fucking insane.
>
> Also, we will lose the history of the Avatica source code. All of the
> commit hashes for bugs fixed long ago will change.
>
> At the time that we split the repositories, I was probably against the
> idea, because it was work. Now that we’ve split, I am strongly against
> recombining the repositories.
>
> Honestly, people, we have better things to do.
>
> Julian
>
>
> > On Nov 9, 2021, at 1:03 PM, Stamatis Zampetakis
> wrote:
> >
> > The two projects have more or less the same release cadence, they're
> > maintained by the same community, they're written in the same language,
> and
> > they use the same build system.
> >
> > Although I understand the reasons the project was split in the first
> place,
> > at the moment I tend to agree that having them in the same repo would be
> > more practical.
> >
> > Truth is the merge may require quite a bit of work so we should take this
> > into consideration.
> >
> > If people prefer keeping the projects as they are I am perfectly fine
> with
> > that as well. I am leaving the decision to those who feel strongly about
> > this change.
> >
> > Best,
> > Stamatis
> >
> >
> >
> > On Tue, Nov 9, 2021, 7:46 PM Vladimir Ozerov wrote:
> >
> >> +1 for a single repo.
> >>
> >> Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov <
> >> sitnikov.vladi...@gmail.com
> >>> :
> >>
> >>> Michael> is not proposing to change the
> >>> Michael>structure of modules within both projects, merely to have the
> >> code
> >>> for
> >>> Michael>both in a single repository.
> >>>
> >>> I propose to integrate them into a single build, and keep the set of
> the
> >>> published jars.
> >>> However, the modules and dependency structure could be kept as is.
> >>>
> >>> We might want to rename folders like
> >>> calcite-core
> >>> calcite-linq4j
> >>> ..
> >>> avatica-core
> >>> avatica-server
> >>>
> >>> However, I am not sure it is that important to discuss folder names
> now.
> >>> The idea is that as you "open Calcite in IDE, you see both Avatica and
> >>> Calcite modules"
> >>>
> >>> Michael>Is there any reason we couldn't have a separate release
> schedule
> >> if
> >>> Michael>both projects are in the same repository?
> >>>
> >>> A different schedule means Calcite must support at least two different
> >>> Avatica versions.
> >>> In other words, if we allow clients to update Avatica at their will,
> then
> >>> we should allow them building Calcite with different Avatica versions,
> >>> which implies Calcite test code should succeed for multiple different
> >>> Avatica vesions.
> >>>
> >>> That makes it harder to write tests: we have to execute tests with two
> >>> different Avatica releases (or even more than two).
> >>>
> >>> There are at least two sources for complexity:
> >>>
> >>> a) We have to write tests that tolerate multiple versions. For
> instance,
> >>> "if (avatica.18+) {...}" and so on.
> >>> That is not really trivial, especially taking into account some of the
> >>> tests are created with non-yet-popular
> >>> technologies like Quidem where you can't really google solutions. So
> the
> >>> "trivial" task of "making a test to expect two possible outcomes"
> >>> becomes less trivial as you try to pass the version from GitHub Action
> to
> >>> Gradle to JUnit to Quidem to no-one-knows-which class.
> >>> If we support one Avatica version only, that is not needed. We just
> patch
> >>> the test in Avatica and Calcite and that is it.
> >>> Single repo avoids "Gradle vs Quidem" dance.
> >>>
> >>> b) If we claim that we support 5 different Guava versions, 3 different
> >> JDK
> >>> versions, 2 different Avatica versions,
> >>> then we have to execute 5*3*2 = 30 combinations of the tests.
> >>> That is not really a full matrix, however, things get way easier if we
> >>> support one Avatica version only.
> >>> The amount of tests we need to do during a proper release is much less,
> >> and
> >>> it is easier to commit
> >>> changes that touch Avatica and Calcite at the same time.
> >>>
> >>>
> >>> Vladimir
> >>>
> >>
>
>
Re: DISCUSS: merge calcite-avatica and calcite repositories
Merging the repositories is unnecessary, and a massive waste of time.
Any supposed “time savings” due to the the merged repositories will be eaten up
by dozens of small issues that will crop up long after the merge.
I was release manager for Calcite and Avatica this time around and it took me
about a week, because the process had changed since last time I was release
manager. And now we’re going to change the process again? Absolutely fucking
insane.
Also, we will lose the history of the Avatica source code. All of the commit
hashes for bugs fixed long ago will change.
At the time that we split the repositories, I was probably against the idea,
because it was work. Now that we’ve split, I am strongly against recombining
the repositories.
Honestly, people, we have better things to do.
Julian
> On Nov 9, 2021, at 1:03 PM, Stamatis Zampetakis wrote:
>
> The two projects have more or less the same release cadence, they're
> maintained by the same community, they're written in the same language, and
> they use the same build system.
>
> Although I understand the reasons the project was split in the first place,
> at the moment I tend to agree that having them in the same repo would be
> more practical.
>
> Truth is the merge may require quite a bit of work so we should take this
> into consideration.
>
> If people prefer keeping the projects as they are I am perfectly fine with
> that as well. I am leaving the decision to those who feel strongly about
> this change.
>
> Best,
> Stamatis
>
>
>
> On Tue, Nov 9, 2021, 7:46 PM Vladimir Ozerov wrote:
>
>> +1 for a single repo.
>>
>> Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov <
>> sitnikov.vladi...@gmail.com
>>> :
>>
>>> Michael> is not proposing to change the
>>> Michael>structure of modules within both projects, merely to have the
>> code
>>> for
>>> Michael>both in a single repository.
>>>
>>> I propose to integrate them into a single build, and keep the set of the
>>> published jars.
>>> However, the modules and dependency structure could be kept as is.
>>>
>>> We might want to rename folders like
>>> calcite-core
>>> calcite-linq4j
>>> ..
>>> avatica-core
>>> avatica-server
>>>
>>> However, I am not sure it is that important to discuss folder names now.
>>> The idea is that as you "open Calcite in IDE, you see both Avatica and
>>> Calcite modules"
>>>
>>> Michael>Is there any reason we couldn't have a separate release schedule
>> if
>>> Michael>both projects are in the same repository?
>>>
>>> A different schedule means Calcite must support at least two different
>>> Avatica versions.
>>> In other words, if we allow clients to update Avatica at their will, then
>>> we should allow them building Calcite with different Avatica versions,
>>> which implies Calcite test code should succeed for multiple different
>>> Avatica vesions.
>>>
>>> That makes it harder to write tests: we have to execute tests with two
>>> different Avatica releases (or even more than two).
>>>
>>> There are at least two sources for complexity:
>>>
>>> a) We have to write tests that tolerate multiple versions. For instance,
>>> "if (avatica.18+) {...}" and so on.
>>> That is not really trivial, especially taking into account some of the
>>> tests are created with non-yet-popular
>>> technologies like Quidem where you can't really google solutions. So the
>>> "trivial" task of "making a test to expect two possible outcomes"
>>> becomes less trivial as you try to pass the version from GitHub Action to
>>> Gradle to JUnit to Quidem to no-one-knows-which class.
>>> If we support one Avatica version only, that is not needed. We just patch
>>> the test in Avatica and Calcite and that is it.
>>> Single repo avoids "Gradle vs Quidem" dance.
>>>
>>> b) If we claim that we support 5 different Guava versions, 3 different
>> JDK
>>> versions, 2 different Avatica versions,
>>> then we have to execute 5*3*2 = 30 combinations of the tests.
>>> That is not really a full matrix, however, things get way easier if we
>>> support one Avatica version only.
>>> The amount of tests we need to do during a proper release is much less,
>> and
>>> it is easier to commit
>>> changes that touch Avatica and Calcite at the same time.
>>>
>>>
>>> Vladimir
>>>
>>
Re: DISCUSS: merge calcite-avatica and calcite repositories
The two projects have more or less the same release cadence, they're
maintained by the same community, they're written in the same language, and
they use the same build system.
Although I understand the reasons the project was split in the first place,
at the moment I tend to agree that having them in the same repo would be
more practical.
Truth is the merge may require quite a bit of work so we should take this
into consideration.
If people prefer keeping the projects as they are I am perfectly fine with
that as well. I am leaving the decision to those who feel strongly about
this change.
Best,
Stamatis
On Tue, Nov 9, 2021, 7:46 PM Vladimir Ozerov wrote:
> +1 for a single repo.
>
> Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov <
> sitnikov.vladi...@gmail.com
> >:
>
> > Michael> is not proposing to change the
> > Michael>structure of modules within both projects, merely to have the
> code
> > for
> > Michael>both in a single repository.
> >
> > I propose to integrate them into a single build, and keep the set of the
> > published jars.
> > However, the modules and dependency structure could be kept as is.
> >
> > We might want to rename folders like
> > calcite-core
> > calcite-linq4j
> > ..
> > avatica-core
> > avatica-server
> >
> > However, I am not sure it is that important to discuss folder names now.
> > The idea is that as you "open Calcite in IDE, you see both Avatica and
> > Calcite modules"
> >
> > Michael>Is there any reason we couldn't have a separate release schedule
> if
> > Michael>both projects are in the same repository?
> >
> > A different schedule means Calcite must support at least two different
> > Avatica versions.
> > In other words, if we allow clients to update Avatica at their will, then
> > we should allow them building Calcite with different Avatica versions,
> > which implies Calcite test code should succeed for multiple different
> > Avatica vesions.
> >
> > That makes it harder to write tests: we have to execute tests with two
> > different Avatica releases (or even more than two).
> >
> > There are at least two sources for complexity:
> >
> > a) We have to write tests that tolerate multiple versions. For instance,
> > "if (avatica.18+) {...}" and so on.
> > That is not really trivial, especially taking into account some of the
> > tests are created with non-yet-popular
> > technologies like Quidem where you can't really google solutions. So the
> > "trivial" task of "making a test to expect two possible outcomes"
> > becomes less trivial as you try to pass the version from GitHub Action to
> > Gradle to JUnit to Quidem to no-one-knows-which class.
> > If we support one Avatica version only, that is not needed. We just patch
> > the test in Avatica and Calcite and that is it.
> > Single repo avoids "Gradle vs Quidem" dance.
> >
> > b) If we claim that we support 5 different Guava versions, 3 different
> JDK
> > versions, 2 different Avatica versions,
> > then we have to execute 5*3*2 = 30 combinations of the tests.
> > That is not really a full matrix, however, things get way easier if we
> > support one Avatica version only.
> > The amount of tests we need to do during a proper release is much less,
> and
> > it is easier to commit
> > changes that touch Avatica and Calcite at the same time.
> >
> >
> > Vladimir
> >
>
Re: DISCUSS: merge calcite-avatica and calcite repositories
+1 for a single repo.
Вт, 9 нояб. 2021 г. в 19:22, Vladimir Sitnikov :
> Michael> is not proposing to change the
> Michael>structure of modules within both projects, merely to have the code
> for
> Michael>both in a single repository.
>
> I propose to integrate them into a single build, and keep the set of the
> published jars.
> However, the modules and dependency structure could be kept as is.
>
> We might want to rename folders like
> calcite-core
> calcite-linq4j
> ..
> avatica-core
> avatica-server
>
> However, I am not sure it is that important to discuss folder names now.
> The idea is that as you "open Calcite in IDE, you see both Avatica and
> Calcite modules"
>
> Michael>Is there any reason we couldn't have a separate release schedule if
> Michael>both projects are in the same repository?
>
> A different schedule means Calcite must support at least two different
> Avatica versions.
> In other words, if we allow clients to update Avatica at their will, then
> we should allow them building Calcite with different Avatica versions,
> which implies Calcite test code should succeed for multiple different
> Avatica vesions.
>
> That makes it harder to write tests: we have to execute tests with two
> different Avatica releases (or even more than two).
>
> There are at least two sources for complexity:
>
> a) We have to write tests that tolerate multiple versions. For instance,
> "if (avatica.18+) {...}" and so on.
> That is not really trivial, especially taking into account some of the
> tests are created with non-yet-popular
> technologies like Quidem where you can't really google solutions. So the
> "trivial" task of "making a test to expect two possible outcomes"
> becomes less trivial as you try to pass the version from GitHub Action to
> Gradle to JUnit to Quidem to no-one-knows-which class.
> If we support one Avatica version only, that is not needed. We just patch
> the test in Avatica and Calcite and that is it.
> Single repo avoids "Gradle vs Quidem" dance.
>
> b) If we claim that we support 5 different Guava versions, 3 different JDK
> versions, 2 different Avatica versions,
> then we have to execute 5*3*2 = 30 combinations of the tests.
> That is not really a full matrix, however, things get way easier if we
> support one Avatica version only.
> The amount of tests we need to do during a proper release is much less, and
> it is easier to commit
> changes that touch Avatica and Calcite at the same time.
>
>
> Vladimir
>
Re: DISCUSS: merge calcite-avatica and calcite repositories
Michael> is not proposing to change the
Michael>structure of modules within both projects, merely to have the code
for
Michael>both in a single repository.
I propose to integrate them into a single build, and keep the set of the
published jars.
However, the modules and dependency structure could be kept as is.
We might want to rename folders like
calcite-core
calcite-linq4j
..
avatica-core
avatica-server
However, I am not sure it is that important to discuss folder names now.
The idea is that as you "open Calcite in IDE, you see both Avatica and
Calcite modules"
Michael>Is there any reason we couldn't have a separate release schedule if
Michael>both projects are in the same repository?
A different schedule means Calcite must support at least two different
Avatica versions.
In other words, if we allow clients to update Avatica at their will, then
we should allow them building Calcite with different Avatica versions,
which implies Calcite test code should succeed for multiple different
Avatica vesions.
That makes it harder to write tests: we have to execute tests with two
different Avatica releases (or even more than two).
There are at least two sources for complexity:
a) We have to write tests that tolerate multiple versions. For instance,
"if (avatica.18+) {...}" and so on.
That is not really trivial, especially taking into account some of the
tests are created with non-yet-popular
technologies like Quidem where you can't really google solutions. So the
"trivial" task of "making a test to expect two possible outcomes"
becomes less trivial as you try to pass the version from GitHub Action to
Gradle to JUnit to Quidem to no-one-knows-which class.
If we support one Avatica version only, that is not needed. We just patch
the test in Avatica and Calcite and that is it.
Single repo avoids "Gradle vs Quidem" dance.
b) If we claim that we support 5 different Guava versions, 3 different JDK
versions, 2 different Avatica versions,
then we have to execute 5*3*2 = 30 combinations of the tests.
That is not really a full matrix, however, things get way easier if we
support one Avatica version only.
The amount of tests we need to do during a proper release is much less, and
it is easier to commit
changes that touch Avatica and Calcite at the same time.
Vladimir
Re: DISCUSS: merge calcite-avatica and calcite repositories
> There are many reasons to divide code into modules. Unless I'm misunderstanding, Vladimir is not proposing to change the structure of modules within both projects, merely to have the code for both in a single repository. > to allow separate release schedules Is there any reason we couldn't have a separate release schedule if both projects are in the same repository? However, it does seem true that if we keep a separate release schedule, the benefits of combining the repositories are less obvious. I don't really have a strong opinion one way or the other, but I think the decision of merging repositories and restructuring code or combing release schedules are two separate decisions. -- Michael Mior mm...@apache.org Le lun. 8 nov. 2021 à 19:43, Julian Hyde a écrit : > > There are many reasons to divide code into modules. To allow separate > communities, to allow separate release schedules, to reduce coupling, to make > it easier to contribute (because contributors don’t need to understand a > large code base), to increase adoption (because people perceive that they can > use component A without using component B). > > The last of these was particularly on my mind when we split Avatica from > Calcite. I was pleased to see, for example, Apache Phoenix using Avatica > successfully (including building an ODBC driver) even though their (separate) > attempt to adopt Calcite failed. > > Splitting code into modules makes it easier to continue to splitting. If > Avatica had remained part of Calcite, both written in Java and using the same > build system and release process, it’s less likely that Avatica-go would have > happened. > > I think the split between Avatica and Calcite repositories is working great. > > Julian > > > > On Nov 8, 2021, at 3:56 PM, Josh Elser wrote: > > > > These repositories are separate because they have separate release > > schedules. Those who were working on calcite.git typically do not want to > > be bothered with changes to calcite-avatica.git, and vice versa. Further, > > there are downstream users of Avatica directly (without Calcite) who would > > be burdened by waiting for a new Calcite release as opposed to a (much more > > simple) Avatica release. > > > > > Recently there was a PR that improves error messages in Avatica: > > > > If I am interpreting your analysis correctly, you are arguing for wanting a > > shorter cycle to "implementing" all parts of a change (both in Avatica and > > Calcite). The net amount of code you have to write doesn't change with how > > it works now compared to how you are suggesting it should work. > > > > If there are breaking changes in Avatica, they would need to be accounted > > for when Calcite is bumped to the next version of Avatica. > > > > In terms of wire compatibility, there has been no breaking wire compat > > change that wasn't due to a problem with the protocol itself (where we had > > to break it). As far as API compatibility goes, I do not believe we have > > ever _had_ to break API compatibility. The untracked runtime changes (like > > the one you cite) are a bigger smell to me in Calcite but that's a > > different conversation to be had. > > > > > * Avatica has fewer commits than Calcite, so having a separate > > > calcite-avatica repository does not help for segregating PR/issue/commit > > > queue > > > > I disagree with your assessment. Having a separate repository makes it > > extremely easy for me to watch Avatica commits/pull-requests which I am > > capable of reviewing vs. Calcite pull-requests which I am not comfortable > > reviewing. > > > > > * calcite-avatica-go seems to reside in its own repository, so I do not > > > see > > > why do we split Java implementations across calcite and calcite-avatica > > > repository > > > > I have no objections to combining these two repositories together. > > > > It seems to me that combining these repositories is one possible solution > > to address the friction of making changes across these two repositories, > > rather than starting from the root problem "How can we make co-dependent > > changes easier?" > > > > I would challenge us instead of complicating releases (which are already > > complicated): what should the API/compatibility surface of Avatica be, and > > what how can be make the gaps you've experienced better? Such a fix will go > > a long way for downstream users of Avatica, too. > > > > - Josh > > > > On 11/8/21 2:39 PM, Vladimir Sitnikov wrote: > >> Hi, > >> Currently, we have calcite-avatica and calcite in different repositories. > >> Frankly speaking, I do not know what it brings, however, it does create > >> points of friction: > >> 1) If a feature touches Avatica and Calcite, then PRs are hard to create > >> and maintain. We just can't create a single PR across both repositories > >> 2) If we support a single Avatica version only in Calcite, then the point > >> of having different repositories is even mooter. > >> 3) CI
Re: DISCUSS: merge calcite-avatica and calcite repositories
Second Vladimir's proposal // Generally speaking，we should decouple the dependence of the two projects on the master branch, Of course, if you merge into one repos, the issue of interdependence does not exist. Best, Jincheng Sun Yogendra Sharma 于2021年11月9日周二 下午3:28写道： > I agree with Vladimir. > > Although i am very new here but in my mind having one repository will > reduce lot of efforts while not necessarily it will reduce flexibility and > modularity of the the project. > > My initial impression is that we can achieve the same modularity > constraints even after merging the code into one repo. > > Get Outlook for Android<https://aka.ms/ghei36> > > > From: Vladimir Sitnikov > Sent: Tuesday, November 9, 2021, 12:38 > To: Apache Calcite dev list > Subject: Re: DISCUSS: merge calcite-avatica and calcite repositories > > I know the below is too wordy, however, English is not my native language, > so I tend to overexplain and duplicate thoughts. > > Julian>To allow separate communities > > This is something I do not understand. > Let me be explicit: I am OK to maintain bits of avatica code when Calcite > fixes overlap with Avatica for some reason (e.g. throwing exceptions). > I am not OK with duplicating the same work just because someone else wants > to have a separate repository. > > I was maintaining and releasing multiple (four!) repositories for pgjdbc > when it was using Maven > (Maven was not able to release jre6, jre7, and jre8 artifacts from within a > single repository), and it was a big relief when the code was recollected > into a single repository again. > > That is why I am advocating for merging avatica code back. > > Julian>Splitting code into modules makes it easier to continue to splitting > > I agree it is worth having calcite-core and avatica as separate jar > artifacts. > It is even worth splitting "enumerable" out of calcite-core into its own > module. > However, I do not see how splitting the repositories helps. I do not expect > somebody to use calcite-avatica as a git submodule or something like that. > > Julian>to allow separate release schedules > > Here are the recent releases. > There's often a pattern that "Calcite release follows Avatica release". > It is even mentioned in the mails where someone suggests releasing Avatica, > then releasing Calcite". > > A notable exception was rel/avatica-1.16.0 which was not connected to the > release of Calcite. > The rest Avatica releases seem to be quite close to Calcite releases, so I > do not see what do we gain from "separate release schedules". > It looks like co-locating the release would be easier for both maintenance > overheads, testing, and consumption. > > > 2016-03-16 rel/calcite-avatica-1.7.0 > > 2016-03-17 calcite-1.7.0 > > 2016-03-15 rel/calcite-avatica-1.7.1 > > 2016-06-01 rel/calcite-avatica-1.8.0 > > 2016-06-08 calcite-1.8.0 > > > 2016-09-17 calcite-1.9.0 > > > 2016-10-07 calcite-1.10.0 > > 2016-10-27 rel/calcite-avatica-1.9.0 > > > 2017-01-03 calcite-1.11.0 > > 2017-03-20 calcite-1.12.0 > > 2017-05-23 rel/avatica-1.10.0 > > 2017-06-22 calcite-1.13.0 > > > 2017-09-27 calcite-1.14.0 > > 2017-12-05 calcite-1.15.0 > > 2018-03-06 rel/avatica-1.11.0 > > 2018-03-14 calcite-1.16.0 > > 2018-06-21 rel/avatica-1.12.0 > > 2018-07-16 calcite-1.17.0 > > 2018-11-29 rel/avatica-1.13.0 > > 2018-12-18 calcite-1.18.0 > > 2019-03-20 calcite-1.19.0 > > 2019-04-25 rel/avatica-1.14.0 > > 2019-05-09 rel/avatica-1.15.0 > > 2019-06-19 calcite-1.20.0 > > > 2019-09-06 calcite-1.21.0 > > 2019-12-19 rel/avatica-1.16.0 > > > 2020-05-23 calcite-1.23.0 > > 2020-06-22 rel/avatica-1.17.0 > > 2020-07-23 calcite-1.24.0 > 2020-08-22 calcite-1.25.0 > > 2020-10-06 calcite-1.26.0 > > > 2021-05-18 rel/avatica-1.18.0 > > 2021-06-03 calcite-1.27.0 > > 2021-10-04 rel/avatica-1.19.0 > > 2021-10-19 calcite-1.28.0 > > Julian>The last of these was particularly on my mind when we split Avatica > from > Calcite. > Julian>I was pleased to see, for example, Apache Phoenix using Avatica > successfully > (including building an ODBC driver) > > Does it really help Phonenix that we split repositories? > I am sure they should not really depend on the repositories. > How would their life be harder if we release the same set of artifacts and > jars from a single source tree? > > Julian>If Avatica had remained part of Calcite, both written in Java and > using the same build system and release process, > Julian>it’s less likely that Avatic
Re: DISCUSS: merge calcite-avatica and calcite repositories
I agree with Vladimir. Although i am very new here but in my mind having one repository will reduce lot of efforts while not necessarily it will reduce flexibility and modularity of the the project. My initial impression is that we can achieve the same modularity constraints even after merging the code into one repo. Get Outlook for Android<https://aka.ms/ghei36> From: Vladimir Sitnikov Sent: Tuesday, November 9, 2021, 12:38 To: Apache Calcite dev list Subject: Re: DISCUSS: merge calcite-avatica and calcite repositories I know the below is too wordy, however, English is not my native language, so I tend to overexplain and duplicate thoughts. Julian>To allow separate communities This is something I do not understand. Let me be explicit: I am OK to maintain bits of avatica code when Calcite fixes overlap with Avatica for some reason (e.g. throwing exceptions). I am not OK with duplicating the same work just because someone else wants to have a separate repository. I was maintaining and releasing multiple (four!) repositories for pgjdbc when it was using Maven (Maven was not able to release jre6, jre7, and jre8 artifacts from within a single repository), and it was a big relief when the code was recollected into a single repository again. That is why I am advocating for merging avatica code back. Julian>Splitting code into modules makes it easier to continue to splitting I agree it is worth having calcite-core and avatica as separate jar artifacts. It is even worth splitting "enumerable" out of calcite-core into its own module. However, I do not see how splitting the repositories helps. I do not expect somebody to use calcite-avatica as a git submodule or something like that. Julian>to allow separate release schedules Here are the recent releases. There's often a pattern that "Calcite release follows Avatica release". It is even mentioned in the mails where someone suggests releasing Avatica, then releasing Calcite". A notable exception was rel/avatica-1.16.0 which was not connected to the release of Calcite. The rest Avatica releases seem to be quite close to Calcite releases, so I do not see what do we gain from "separate release schedules". It looks like co-locating the release would be easier for both maintenance overheads, testing, and consumption. 2016-03-16 rel/calcite-avatica-1.7.0 2016-03-17 calcite-1.7.0 2016-03-15 rel/calcite-avatica-1.7.1 2016-06-01 rel/calcite-avatica-1.8.0 2016-06-08 calcite-1.8.0 2016-09-17 calcite-1.9.0 2016-10-07 calcite-1.10.0 2016-10-27 rel/calcite-avatica-1.9.0 2017-01-03 calcite-1.11.0 2017-03-20 calcite-1.12.0 2017-05-23 rel/avatica-1.10.0 2017-06-22 calcite-1.13.0 2017-09-27 calcite-1.14.0 2017-12-05 calcite-1.15.0 2018-03-06 rel/avatica-1.11.0 2018-03-14 calcite-1.16.0 2018-06-21 rel/avatica-1.12.0 2018-07-16 calcite-1.17.0 2018-11-29 rel/avatica-1.13.0 2018-12-18 calcite-1.18.0 2019-03-20 calcite-1.19.0 2019-04-25 rel/avatica-1.14.0 2019-05-09 rel/avatica-1.15.0 2019-06-19 calcite-1.20.0 2019-09-06 calcite-1.21.0 2019-12-19 rel/avatica-1.16.0 2020-05-23 calcite-1.23.0 2020-06-22 rel/avatica-1.17.0 2020-07-23 calcite-1.24.0 2020-08-22 calcite-1.25.0 2020-10-06 calcite-1.26.0 2021-05-18 rel/avatica-1.18.0 2021-06-03 calcite-1.27.0 2021-10-04 rel/avatica-1.19.0 2021-10-19 calcite-1.28.0 Julian>The last of these was particularly on my mind when we split Avatica from Calcite. Julian>I was pleased to see, for example, Apache Phoenix using Avatica successfully (including building an ODBC driver) Does it really help Phonenix that we split repositories? I am sure they should not really depend on the repositories. How would their life be harder if we release the same set of artifacts and jars from a single source tree? Julian>If Avatica had remained part of Calcite, both written in Java and using the same build system and release process, Julian>it’s less likely that Avatica-go would have happened Does Avatica-go use calcite-avatica repository? So far it looks like Avatica-go uses https://github.com/apache/calcite-avatica-go, and I do not suggest merging it into calcite repository yet. I do not see how merging calcite-avatica and calcite repositories would block the evolution or emergence of avatica-go, avatica-rust, etc. Julian>release process I identified https://issues.apache.org/jira/browse/CALCITE-4881 "Calcite release tags should have rel/... prefix as per the ASF requirements" It looks like an unintentional breakage in https://github.com/apache/calcite/commit/2e30293af7373b6c5fbcc5fa6505b49df2fba000 caused by the split. Vladimir
Re: DISCUSS: merge calcite-avatica and calcite repositories
Francis>I remembered joining this list a few years back then Francis>- Calcite - SQL engine used to build the SQL part of your database. The things is "apache/calcite" is tons of features as in "thing that converts SQL to MongoDB" or "thing that executes SQL queries over InnoDB data files", or "thing that validates SQL", or "linq4j". There are lots of features, and there are lots of artifacts. The set of modules is not perfect, and there are cases when somebody needs "just optimizer", and there are valid reasons to hate "Enumerable" or "dynamic bytecode generation". However, it looks like having Avatica in the same repository does not hurt much as long as the other features do not impact Avatica features much. Francis>Each should be able to stand up on its own and have its own independent release schedule Having a different release schedule means we must support at least two different Avatica versions in Calcite. That adds overhead on Calcite development as we have to make test code to account for different Avatica. Francis>I've seen PRs that need to touch both repositories for Francis>a change and it seems like a code smell to me Even though testing the "exact exception message" at Calcite side looks weird, it is important to test how the exception would look like for the end-user, so there must be an integration test that ensures Avatica produces proper exceptions, and Calcite does not make them bad (e.g. by ignoring the message). Francis>own independent release schedule For now, Calcite Avatica is stuck with Gradle 6 since it uses "dependencies between test source sets". In other words, test classes in calcite-avatica/server depend on test classes in calcite-avatica/core. That is an anti-pattern, and the resolution is to move the common test classes into a proper artifact with its metadata and dependencies. This is something I fixed in Calcite in https://issues.apache.org/jira/browse/CALCITE-4821 Here's the question: who would do the relevant work for calcite-avatica? Vladimir
Re: DISCUSS: merge calcite-avatica and calcite repositories
I know the below is too wordy, however, English is not my native language, so I tend to overexplain and duplicate thoughts. Julian>To allow separate communities This is something I do not understand. Let me be explicit: I am OK to maintain bits of avatica code when Calcite fixes overlap with Avatica for some reason (e.g. throwing exceptions). I am not OK with duplicating the same work just because someone else wants to have a separate repository. I was maintaining and releasing multiple (four!) repositories for pgjdbc when it was using Maven (Maven was not able to release jre6, jre7, and jre8 artifacts from within a single repository), and it was a big relief when the code was recollected into a single repository again. That is why I am advocating for merging avatica code back. Julian>Splitting code into modules makes it easier to continue to splitting I agree it is worth having calcite-core and avatica as separate jar artifacts. It is even worth splitting "enumerable" out of calcite-core into its own module. However, I do not see how splitting the repositories helps. I do not expect somebody to use calcite-avatica as a git submodule or something like that. Julian>to allow separate release schedules Here are the recent releases. There's often a pattern that "Calcite release follows Avatica release". It is even mentioned in the mails where someone suggests releasing Avatica, then releasing Calcite". A notable exception was rel/avatica-1.16.0 which was not connected to the release of Calcite. The rest Avatica releases seem to be quite close to Calcite releases, so I do not see what do we gain from "separate release schedules". It looks like co-locating the release would be easier for both maintenance overheads, testing, and consumption. 2016-03-16 rel/calcite-avatica-1.7.0 2016-03-17 calcite-1.7.0 2016-03-15 rel/calcite-avatica-1.7.1 2016-06-01 rel/calcite-avatica-1.8.0 2016-06-08 calcite-1.8.0 2016-09-17 calcite-1.9.0 2016-10-07 calcite-1.10.0 2016-10-27 rel/calcite-avatica-1.9.0 2017-01-03 calcite-1.11.0 2017-03-20 calcite-1.12.0 2017-05-23 rel/avatica-1.10.0 2017-06-22 calcite-1.13.0 2017-09-27 calcite-1.14.0 2017-12-05 calcite-1.15.0 2018-03-06 rel/avatica-1.11.0 2018-03-14 calcite-1.16.0 2018-06-21 rel/avatica-1.12.0 2018-07-16 calcite-1.17.0 2018-11-29 rel/avatica-1.13.0 2018-12-18 calcite-1.18.0 2019-03-20 calcite-1.19.0 2019-04-25 rel/avatica-1.14.0 2019-05-09 rel/avatica-1.15.0 2019-06-19 calcite-1.20.0 2019-09-06 calcite-1.21.0 2019-12-19 rel/avatica-1.16.0 2020-05-23 calcite-1.23.0 2020-06-22 rel/avatica-1.17.0 2020-07-23 calcite-1.24.0 2020-08-22 calcite-1.25.0 2020-10-06 calcite-1.26.0 2021-05-18 rel/avatica-1.18.0 2021-06-03 calcite-1.27.0 2021-10-04 rel/avatica-1.19.0 2021-10-19 calcite-1.28.0 Julian>The last of these was particularly on my mind when we split Avatica from Calcite. Julian>I was pleased to see, for example, Apache Phoenix using Avatica successfully (including building an ODBC driver) Does it really help Phonenix that we split repositories? I am sure they should not really depend on the repositories. How would their life be harder if we release the same set of artifacts and jars from a single source tree? Julian>If Avatica had remained part of Calcite, both written in Java and using the same build system and release process, Julian>it’s less likely that Avatica-go would have happened Does Avatica-go use calcite-avatica repository? So far it looks like Avatica-go uses https://github.com/apache/calcite-avatica-go, and I do not suggest merging it into calcite repository yet. I do not see how merging calcite-avatica and calcite repositories would block the evolution or emergence of avatica-go, avatica-rust, etc. Julian>release process I identified https://issues.apache.org/jira/browse/CALCITE-4881 "Calcite release tags should have rel/... prefix as per the ASF requirements" It looks like an unintentional breakage in https://github.com/apache/calcite/commit/2e30293af7373b6c5fbcc5fa6505b49df2fba000 caused by the split. Vladimir
Re: DISCUSS: merge calcite-avatica and calcite repositories
I think it is still worthwhile to keep Avatica and Calcite as 2 separate repositories. I remembered joining this list a few years back then Avatica was moved into its own repository and my perception of the components were: - Avatica - HTTP server that you can put in front of your database so that you can easily write or generate a client in any language to talk to the database. - Calcite - SQL engine used to build the SQL part of your database. From this perspective, it does not seem that the 2 components should be so tightly coupled and be in the same repository. Each should be able to stand up on its own and have its own independent release schedule. Having said that, I've seen PRs that need to touch both repositories for a change and it seems like a code smell to me. Perhaps we should look at why those 2 seemly separate components are tightly coupled together and what we can do to remove the coupling. Maybe this means moving some code from Avatica to Calcite or vice-versa to severe the coupling so that both components are truly independent and a change won't need to touch both components. On 9/11/2021 12:10 pm, xiong duan wrote: I have no opinion on merging the two repositories or don't. But If we don't merge it. About CALCITE-4877, Maybe we can: 1) we cancel test Calcite's master branch with Avatica's master branch directly. (It is unreasonable) When have an ISSUE need to submit PR in both, we can try: 1) If the PR is good. We can merge the PR in AVATICA at any time(Because it is a new version). 2) But The PR for Calcite, *needs to include the AVATICA new's version*(not released but can ensure run CI jobs in this PR to make sure the PR is good). 3)Until the Calcite prepares to upgrade the dependency Avatica version, Then we can merge it. (As the last version, We create a Jira to log which PR needs to merge). Julian Hyde 于2021年11月9日周二 上午8:43写道： There are many reasons to divide code into modules. To allow separate communities, to allow separate release schedules, to reduce coupling, to make it easier to contribute (because contributors don’t need to understand a large code base), to increase adoption (because people perceive that they can use component A without using component B). The last of these was particularly on my mind when we split Avatica from Calcite. I was pleased to see, for example, Apache Phoenix using Avatica successfully (including building an ODBC driver) even though their (separate) attempt to adopt Calcite failed. Splitting code into modules makes it easier to continue to splitting. If Avatica had remained part of Calcite, both written in Java and using the same build system and release process, it’s less likely that Avatica-go would have happened. I think the split between Avatica and Calcite repositories is working great. Julian On Nov 8, 2021, at 3:56 PM, Josh Elser wrote: These repositories are separate because they have separate release schedules. Those who were working on calcite.git typically do not want to be bothered with changes to calcite-avatica.git, and vice versa. Further, there are downstream users of Avatica directly (without Calcite) who would be burdened by waiting for a new Calcite release as opposed to a (much more simple) Avatica release. Recently there was a PR that improves error messages in Avatica: If I am interpreting your analysis correctly, you are arguing for wanting a shorter cycle to "implementing" all parts of a change (both in Avatica and Calcite). The net amount of code you have to write doesn't change with how it works now compared to how you are suggesting it should work. If there are breaking changes in Avatica, they would need to be accounted for when Calcite is bumped to the next version of Avatica. In terms of wire compatibility, there has been no breaking wire compat change that wasn't due to a problem with the protocol itself (where we had to break it). As far as API compatibility goes, I do not believe we have ever _had_ to break API compatibility. The untracked runtime changes (like the one you cite) are a bigger smell to me in Calcite but that's a different conversation to be had. * Avatica has fewer commits than Calcite, so having a separate calcite-avatica repository does not help for segregating PR/issue/commit queue I disagree with your assessment. Having a separate repository makes it extremely easy for me to watch Avatica commits/pull-requests which I am capable of reviewing vs. Calcite pull-requests which I am not comfortable reviewing. * calcite-avatica-go seems to reside in its own repository, so I do not see why do we split Java implementations across calcite and calcite-avatica repository I have no objections to combining these two repositories together. It seems to me that combining these repositories is one possible solution to address the friction of making changes across these two repositories, rather than starting from the root problem "How can we make
Re: DISCUSS: merge calcite-avatica and calcite repositories
I have no opinion on merging the two repositories or don't. But If we don't merge it. About CALCITE-4877, Maybe we can: 1) we cancel test Calcite's master branch with Avatica's master branch directly. (It is unreasonable) When have an ISSUE need to submit PR in both, we can try: 1) If the PR is good. We can merge the PR in AVATICA at any time(Because it is a new version). 2) But The PR for Calcite, *needs to include the AVATICA new's version*(not released but can ensure run CI jobs in this PR to make sure the PR is good). 3)Until the Calcite prepares to upgrade the dependency Avatica version, Then we can merge it. (As the last version, We create a Jira to log which PR needs to merge). Julian Hyde 于2021年11月9日周二 上午8:43写道： > There are many reasons to divide code into modules. To allow separate > communities, to allow separate release schedules, to reduce coupling, to > make it easier to contribute (because contributors don’t need to understand > a large code base), to increase adoption (because people perceive that they > can use component A without using component B). > > The last of these was particularly on my mind when we split Avatica from > Calcite. I was pleased to see, for example, Apache Phoenix using Avatica > successfully (including building an ODBC driver) even though their > (separate) attempt to adopt Calcite failed. > > Splitting code into modules makes it easier to continue to splitting. If > Avatica had remained part of Calcite, both written in Java and using the > same build system and release process, it’s less likely that Avatica-go > would have happened. > > I think the split between Avatica and Calcite repositories is working > great. > > Julian > > > > On Nov 8, 2021, at 3:56 PM, Josh Elser wrote: > > > > These repositories are separate because they have separate release > schedules. Those who were working on calcite.git typically do not want to > be bothered with changes to calcite-avatica.git, and vice versa. Further, > there are downstream users of Avatica directly (without Calcite) who would > be burdened by waiting for a new Calcite release as opposed to a (much more > simple) Avatica release. > > > > > Recently there was a PR that improves error messages in Avatica: > > > > If I am interpreting your analysis correctly, you are arguing for > wanting a shorter cycle to "implementing" all parts of a change (both in > Avatica and Calcite). The net amount of code you have to write doesn't > change with how it works now compared to how you are suggesting it should > work. > > > > If there are breaking changes in Avatica, they would need to be > accounted for when Calcite is bumped to the next version of Avatica. > > > > In terms of wire compatibility, there has been no breaking wire compat > change that wasn't due to a problem with the protocol itself (where we had > to break it). As far as API compatibility goes, I do not believe we have > ever _had_ to break API compatibility. The untracked runtime changes (like > the one you cite) are a bigger smell to me in Calcite but that's a > different conversation to be had. > > > > > * Avatica has fewer commits than Calcite, so having a separate > > > calcite-avatica repository does not help for segregating > PR/issue/commit > > > queue > > > > I disagree with your assessment. Having a separate repository makes it > extremely easy for me to watch Avatica commits/pull-requests which I am > capable of reviewing vs. Calcite pull-requests which I am not comfortable > reviewing. > > > > > * calcite-avatica-go seems to reside in its own repository, so I do > not see > > > why do we split Java implementations across calcite and calcite-avatica > > > repository > > > > I have no objections to combining these two repositories together. > > > > It seems to me that combining these repositories is one possible > solution to address the friction of making changes across these two > repositories, rather than starting from the root problem "How can we make > co-dependent changes easier?" > > > > I would challenge us instead of complicating releases (which are already > complicated): what should the API/compatibility surface of Avatica be, and > what how can be make the gaps you've experienced better? Such a fix will go > a long way for downstream users of Avatica, too. > > > > - Josh > > > > On 11/8/21 2:39 PM, Vladimir Sitnikov wrote: > >> Hi, > >> Currently, we have calcite-avatica and calcite in different > repositories. > >> Frankly speaking, I do not know what it brings, however, it does create > >> points of friction: > >> 1) If a feature touches Avatica and Calcite, then PRs are hard to create > >> and maintain. We just can't create a single PR across both repositories > >> 2) If we support a single Avatica version only in Calcite, then the > point > >> of having different repositories is even mooter. > >> 3) CI configuration is basically duplicated: every time we want to add a > >> new JDK (once every 6 times), we have to do it twice > >> 4)
Re: DISCUSS: merge calcite-avatica and calcite repositories
There are many reasons to divide code into modules. To allow separate communities, to allow separate release schedules, to reduce coupling, to make it easier to contribute (because contributors don’t need to understand a large code base), to increase adoption (because people perceive that they can use component A without using component B). The last of these was particularly on my mind when we split Avatica from Calcite. I was pleased to see, for example, Apache Phoenix using Avatica successfully (including building an ODBC driver) even though their (separate) attempt to adopt Calcite failed. Splitting code into modules makes it easier to continue to splitting. If Avatica had remained part of Calcite, both written in Java and using the same build system and release process, it’s less likely that Avatica-go would have happened. I think the split between Avatica and Calcite repositories is working great. Julian > On Nov 8, 2021, at 3:56 PM, Josh Elser wrote: > > These repositories are separate because they have separate release schedules. > Those who were working on calcite.git typically do not want to be bothered > with changes to calcite-avatica.git, and vice versa. Further, there are > downstream users of Avatica directly (without Calcite) who would be burdened > by waiting for a new Calcite release as opposed to a (much more simple) > Avatica release. > > > Recently there was a PR that improves error messages in Avatica: > > If I am interpreting your analysis correctly, you are arguing for wanting a > shorter cycle to "implementing" all parts of a change (both in Avatica and > Calcite). The net amount of code you have to write doesn't change with how it > works now compared to how you are suggesting it should work. > > If there are breaking changes in Avatica, they would need to be accounted for > when Calcite is bumped to the next version of Avatica. > > In terms of wire compatibility, there has been no breaking wire compat change > that wasn't due to a problem with the protocol itself (where we had to break > it). As far as API compatibility goes, I do not believe we have ever _had_ to > break API compatibility. The untracked runtime changes (like the one you > cite) are a bigger smell to me in Calcite but that's a different conversation > to be had. > > > * Avatica has fewer commits than Calcite, so having a separate > > calcite-avatica repository does not help for segregating PR/issue/commit > > queue > > I disagree with your assessment. Having a separate repository makes it > extremely easy for me to watch Avatica commits/pull-requests which I am > capable of reviewing vs. Calcite pull-requests which I am not comfortable > reviewing. > > > * calcite-avatica-go seems to reside in its own repository, so I do not see > > why do we split Java implementations across calcite and calcite-avatica > > repository > > I have no objections to combining these two repositories together. > > It seems to me that combining these repositories is one possible solution to > address the friction of making changes across these two repositories, rather > than starting from the root problem "How can we make co-dependent changes > easier?" > > I would challenge us instead of complicating releases (which are already > complicated): what should the API/compatibility surface of Avatica be, and > what how can be make the gaps you've experienced better? Such a fix will go a > long way for downstream users of Avatica, too. > > - Josh > > On 11/8/21 2:39 PM, Vladimir Sitnikov wrote: >> Hi, >> Currently, we have calcite-avatica and calcite in different repositories. >> Frankly speaking, I do not know what it brings, however, it does create >> points of friction: >> 1) If a feature touches Avatica and Calcite, then PRs are hard to create >> and maintain. We just can't create a single PR across both repositories >> 2) If we support a single Avatica version only in Calcite, then the point >> of having different repositories is even mooter. >> 3) CI configuration is basically duplicated: every time we want to add a >> new JDK (once every 6 times), we have to do it twice >> 4) There are common dependencies: JUnit, hamcrest, etc, etc. We basically >> have to do the same thing multiple times when upgrading versions in avatica >> and calcite >> 5) Adding @Nullable annotations to Calcite was more complicated than I >> wanted because Avatica is stored in a different repository. >> I basically had to create a bunch of astub files instead of just putting >> the relevant @nullable annotations on top of Avatica classes: >> https://github.com/apache/calcite/tree/f1db79fb876ac9ba3c405283e99bb0438e4e97be/src/main/config/checkerframework/avatica >> Recently there was a PR that improves error messages in Avatica: >> https://github.com/apache/calcite-avatica/pull/161 >> I am sure the PR is a great improvement, however, it fails CI in both cases: >> a) Current Avatica fails when it runs integration
Re: DISCUSS: merge calcite-avatica and calcite repositories
These repositories are separate because they have separate release schedules. Those who were working on calcite.git typically do not want to be bothered with changes to calcite-avatica.git, and vice versa. Further, there are downstream users of Avatica directly (without Calcite) who would be burdened by waiting for a new Calcite release as opposed to a (much more simple) Avatica release. > Recently there was a PR that improves error messages in Avatica: If I am interpreting your analysis correctly, you are arguing for wanting a shorter cycle to "implementing" all parts of a change (both in Avatica and Calcite). The net amount of code you have to write doesn't change with how it works now compared to how you are suggesting it should work. If there are breaking changes in Avatica, they would need to be accounted for when Calcite is bumped to the next version of Avatica. In terms of wire compatibility, there has been no breaking wire compat change that wasn't due to a problem with the protocol itself (where we had to break it). As far as API compatibility goes, I do not believe we have ever _had_ to break API compatibility. The untracked runtime changes (like the one you cite) are a bigger smell to me in Calcite but that's a different conversation to be had. > * Avatica has fewer commits than Calcite, so having a separate > calcite-avatica repository does not help for segregating PR/issue/commit > queue I disagree with your assessment. Having a separate repository makes it extremely easy for me to watch Avatica commits/pull-requests which I am capable of reviewing vs. Calcite pull-requests which I am not comfortable reviewing. > * calcite-avatica-go seems to reside in its own repository, so I do not see > why do we split Java implementations across calcite and calcite-avatica > repository I have no objections to combining these two repositories together. It seems to me that combining these repositories is one possible solution to address the friction of making changes across these two repositories, rather than starting from the root problem "How can we make co-dependent changes easier?" I would challenge us instead of complicating releases (which are already complicated): what should the API/compatibility surface of Avatica be, and what how can be make the gaps you've experienced better? Such a fix will go a long way for downstream users of Avatica, too. - Josh On 11/8/21 2:39 PM, Vladimir Sitnikov wrote: Hi, Currently, we have calcite-avatica and calcite in different repositories. Frankly speaking, I do not know what it brings, however, it does create points of friction: 1) If a feature touches Avatica and Calcite, then PRs are hard to create and maintain. We just can't create a single PR across both repositories 2) If we support a single Avatica version only in Calcite, then the point of having different repositories is even mooter. 3) CI configuration is basically duplicated: every time we want to add a new JDK (once every 6 times), we have to do it twice 4) There are common dependencies: JUnit, hamcrest, etc, etc. We basically have to do the same thing multiple times when upgrading versions in avatica and calcite 5) Adding @Nullable annotations to Calcite was more complicated than I wanted because Avatica is stored in a different repository. I basically had to create a bunch of astub files instead of just putting the relevant @nullable annotations on top of Avatica classes: https://github.com/apache/calcite/tree/f1db79fb876ac9ba3c405283e99bb0438e4e97be/src/main/config/checkerframework/avatica Recently there was a PR that improves error messages in Avatica: https://github.com/apache/calcite-avatica/pull/161 I am sure the PR is a great improvement, however, it fails CI in both cases: a) Current Avatica fails when it runs integration tests against Calcite (because Calcite expects old, low-detail exception messages) b) Current Calcite fails to build with "latest Avatica" because, well, Avatica produces "too good" exception messages It surfaces a true problem: we have too tight code integration between "different" systems, and it probably makes sense to have both libraries in a single repository. An alternative option is to make sure Calcite "supports" at least two Avatica versions: "previous version + one new". However, the current tests in Calcite expect a specific error message, so it can't support two alternative messges. Well, the tests are in .iq format which could probably support multiple messages, however, I have absolutely no idea how to implement that. Facts so far: * Avatica has fewer commits than Calcite, so having a separate calcite-avatica repository does not help for segregating PR/issue/commit queue * Calcite seems to support one specific Avatica version only, so it makes sense to just keep them in a single repository * calcite-avatica-go seems to reside in its own repository, so I do not see why do we split Java implementations across
Re: DISCUSS: merge calcite-avatica and calcite repositories
Currently CI for avatica verifies building with calcite/master, and vice versa. Otherwise we won't notice integration issues before we release. Julian suggested that we should support one avatica version only. I agree, especially of that means we merge the repositories back. > I think it is a mistake to build against master We expect certain level of backward compatibility, so there are reasons to build with latest commits. However, if the versions are always aligned in practice, then having a single repo seems reasonable. One more solution is to disable/remove the offending test in Calcite. Merging the repos still make sense for me, hovewer, I might miss some use cases. Vladimir
Re: DISCUSS: merge calcite-avatica and calcite repositories
Can you confirm that Calcite is building against both avatica latest release and avatica master (this seems to be the case based on an avatica merge breaking calcite ci)? If we truly want to treat the projects as separate, I would suggest that Calcite only be built against released avatica artifacts. If we're building master calcite against master avatica, I think mono repo makes sense but I think it is a mistake to build against master. This basically reinforces the coupling. On Mon, Nov 8, 2021 at 11:39 AM Vladimir Sitnikov < sitnikov.vladi...@gmail.com> wrote: > Hi, > > Currently, we have calcite-avatica and calcite in different repositories. > Frankly speaking, I do not know what it brings, however, it does create > points of friction: > 1) If a feature touches Avatica and Calcite, then PRs are hard to create > and maintain. We just can't create a single PR across both repositories > 2) If we support a single Avatica version only in Calcite, then the point > of having different repositories is even mooter. > 3) CI configuration is basically duplicated: every time we want to add a > new JDK (once every 6 times), we have to do it twice > 4) There are common dependencies: JUnit, hamcrest, etc, etc. We basically > have to do the same thing multiple times when upgrading versions in avatica > and calcite > 5) Adding @Nullable annotations to Calcite was more complicated than I > wanted because Avatica is stored in a different repository. > I basically had to create a bunch of astub files instead of just putting > the relevant @nullable annotations on top of Avatica classes: > > https://github.com/apache/calcite/tree/f1db79fb876ac9ba3c405283e99bb0438e4e97be/src/main/config/checkerframework/avatica > > Recently there was a PR that improves error messages in Avatica: > https://github.com/apache/calcite-avatica/pull/161 > I am sure the PR is a great improvement, however, it fails CI in both > cases: > a) Current Avatica fails when it runs integration tests against Calcite > (because Calcite expects old, low-detail exception messages) > b) Current Calcite fails to build with "latest Avatica" because, well, > Avatica produces "too good" exception messages > > It surfaces a true problem: we have too tight code integration between > "different" systems, and it probably makes sense to have both libraries in > a single repository. > > An alternative option is to make sure Calcite "supports" at least two > Avatica versions: "previous version + one new". > > However, the current tests in Calcite expect a specific error message, so > it can't support two alternative messges. > Well, the tests are in .iq format which could probably support multiple > messages, however, I have absolutely no idea how to implement that. > > Facts so far: > > * Avatica has fewer commits than Calcite, so having a separate > calcite-avatica repository does not help for segregating PR/issue/commit > queue > * Calcite seems to support one specific Avatica version only, so it makes > sense to just keep them in a single repository > * calcite-avatica-go seems to reside in its own repository, so I do not see > why do we split Java implementations across calcite and calcite-avatica > repository > * There is non-trivial maintenance overhead (see 1..5 above). Frankly > speaking, I was trying my best to **avoid** maintaining calcite-avatica. > Somebody wanted to go into a separate repository, so, I let them do what > they want there. > However, there are cases when I have to spend extra time because > calcite-avatica is a separate repository (PR161, @Nullable are the recent > samples) > * It looks like I broke the build by merging PR#161. That is why I am > trying to roll the thing forward and bring this discussion. > An alternative option is I revert the merge and wait for somebody else to > pick up the task. > > So my questions are: > > Q1) Does having calcite-avatica as a separate repository do anybody any > good? > Q2) Does anybody object to merging calcite-avatica and calcite into a > single calcite repository? > > Vladimir >
DISCUSS: merge calcite-avatica and calcite repositories
Hi, Currently, we have calcite-avatica and calcite in different repositories. Frankly speaking, I do not know what it brings, however, it does create points of friction: 1) If a feature touches Avatica and Calcite, then PRs are hard to create and maintain. We just can't create a single PR across both repositories 2) If we support a single Avatica version only in Calcite, then the point of having different repositories is even mooter. 3) CI configuration is basically duplicated: every time we want to add a new JDK (once every 6 times), we have to do it twice 4) There are common dependencies: JUnit, hamcrest, etc, etc. We basically have to do the same thing multiple times when upgrading versions in avatica and calcite 5) Adding @Nullable annotations to Calcite was more complicated than I wanted because Avatica is stored in a different repository. I basically had to create a bunch of astub files instead of just putting the relevant @nullable annotations on top of Avatica classes: https://github.com/apache/calcite/tree/f1db79fb876ac9ba3c405283e99bb0438e4e97be/src/main/config/checkerframework/avatica Recently there was a PR that improves error messages in Avatica: https://github.com/apache/calcite-avatica/pull/161 I am sure the PR is a great improvement, however, it fails CI in both cases: a) Current Avatica fails when it runs integration tests against Calcite (because Calcite expects old, low-detail exception messages) b) Current Calcite fails to build with "latest Avatica" because, well, Avatica produces "too good" exception messages It surfaces a true problem: we have too tight code integration between "different" systems, and it probably makes sense to have both libraries in a single repository. An alternative option is to make sure Calcite "supports" at least two Avatica versions: "previous version + one new". However, the current tests in Calcite expect a specific error message, so it can't support two alternative messges. Well, the tests are in .iq format which could probably support multiple messages, however, I have absolutely no idea how to implement that. Facts so far: * Avatica has fewer commits than Calcite, so having a separate calcite-avatica repository does not help for segregating PR/issue/commit queue * Calcite seems to support one specific Avatica version only, so it makes sense to just keep them in a single repository * calcite-avatica-go seems to reside in its own repository, so I do not see why do we split Java implementations across calcite and calcite-avatica repository * There is non-trivial maintenance overhead (see 1..5 above). Frankly speaking, I was trying my best to **avoid** maintaining calcite-avatica. Somebody wanted to go into a separate repository, so, I let them do what they want there. However, there are cases when I have to spend extra time because calcite-avatica is a separate repository (PR161, @Nullable are the recent samples) * It looks like I broke the build by merging PR#161. That is why I am trying to roll the thing forward and bring this discussion. An alternative option is I revert the merge and wait for somebody else to pick up the task. So my questions are: Q1) Does having calcite-avatica as a separate repository do anybody any good? Q2) Does anybody object to merging calcite-avatica and calcite into a single calcite repository? Vladimir
