Hi all, Thanks for the fruitful discussion. It seems there is not a "single formula" that is safe for all scenarios.
In addition, it requires some effort to recognize the particular scenario we are in (commits out of order, missing commits, etc.). Sometimes it is not reliable to simply check the commit message to determine if two commits have the same contents. So it would be great if we can simply work on a single branch. Best, Liya Fan Francis Chuang <[email protected]> 于2022年3月28日周一 18:06写道： > > If all version specific documentation is siloed into their own > respective folders for each version, then this will be much easier to > automate as we can just simply build and publish the site on every push > to master. > > Each version would be in each folder, with the unreleased version being > in the devel folder. > > Currently, I think the documentation is scattered across various folders > and files in the website, so to simply build from every commit to master > would mean that users will see stuff only relevant to the unreleased > version. This may or may not be confusing. > > We can use some heuristics as I mentioned earlier: keep the current way > but automate it. > - Have a github action that watches every commit, if it only touches a > list of whitelisted folders or pages that we know will not be > documentation for a future release (news, community, etc), cherry pick > it to the site branch > - Have a github action that watches for a final release and force site > to equal master > - On every commit to the site branch, build and publish the site > > Francis > > On 28/03/2022 8:55 pm, Stamatis Zampetakis wrote: > > Having multiple APIs versions in the website has been discussed here [1]. > > > > Since this work of automation is important and has been postponed many > > times in the past I think it is important to get something simple to begin > > with and add "new features" like versioned documentation later on and if > > there is interest. > > > > Building the site from the master on every commit is as simple as it can > > get and that's why I brought it up. > > Having said that, any other option which goes one step further gets a +1 > > from me. > > > > Best, > > Stamatis > > > > [1] https://lists.apache.org/thread/l81th3qvdwttgk135nplz983m78d62m7 > > > > On Mon, Mar 28, 2022 at 11:38 AM Ruben Q L <[email protected]> wrote: > > > >> Would it be clearer if we had different API versions on the site? > >> We could have one API link per Calcite version (or at least for the latest > >> X versions) + an API link of the current master head (that could be updated > >> automatically). > >> I think this "multiple API" idea has been already discussed in the past, > >> but I could not find the thread. > >> > >> > >> > >> On Mon, Mar 28, 2022 at 12:24 AM Francis Chuang <[email protected]> > >> wrote: > >> > >>> It looks like Infra should be able to give us a token to push to > >>> calcite-site from our other calcite-* repos using Github actions [1]. > >>> > >>> If we can have some consensus regarding whether to keep the site branch > >>> and maintain the current process, or to remove it and just publish from > >>> master, I can see if I can get the automated site builds moving along. > >>> > >>> [1] https://issues.apache.org/jira/browse/INFRA-21453 > >>> > >>> On 28/03/2022 8:23 am, Stamatis Zampetakis wrote: > >>>> It would be great if we manage to wrap up CALCITE-3129 and have an > >>>> automated build for the website. > >>>> > >>>> The thing that complicates the procedure in general (automated or not) > >> is > >>>> the fact that we don't want to publish API related changes on the web > >>>> before they are officially released. > >>>> I understand the benefits for trying to maintain this practice but I > >>> would > >>>> be willing to sacrifice those for having simpler procedures/scripts. > >>>> > >>>> I rarely search for any javadoc (Calcite or other) online because > >>> whenever > >>>> I need something the IDE fetches it for me. In most cases, the javadoc > >>>> won't be enough and I will need to dig in the code which is again > >> fetched > >>>> automatically by IDE. If nothing works, and the project is open > >> source, I > >>>> will simply download the respective project and look into the > >>> javadoc/code > >>>> directly. > >>>> > >>>> Apart from that, users will rarely jump to the latest Calcite version > >>>> directly so having the corresponding javadoc online might not be very > >>>> helpful. > >>>> > >>>> Long story short, another option would be to build/update the website > >>>> directly after every commit on master or at certain intervals (e.g., > >>> daily) > >>>> and not have any other branches to maintain. > >>>> > >>>> If there are really people using the published javadoc on the website > >>> [1], > >>>> I would really like to hear their thoughts about this proposal. > >>>> > >>>> Best, > >>>> Stamatis > >>>> > >>>> [1] https://calcite.apache.org/javadocAggregate/ > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On Sat, Mar 26, 2022 at 10:57 PM Francis Chuang < > >>> [email protected]> > >>>> wrote: > >>>> > >>>>> Ideally, I would like to see that the site builds are automated by CI, > >>>>> we still have CALCITE-3129 [1] open. > >>>>> > >>>>> My thinking is that if we automate the site building and deployment > >>>>> process, we can use the following heuristics: > >>>>> - Build the site completely and deploy when a final release tag is > >>>>> pushed to the repo. > >>>>> - Build the site on a partial basis in all other cases: > >>>>> - Option 1: Check out the last final release tag and apply changes > >>> to > >>>>> the site that only touches certain whitelisted categories such as news > >>>>> and community. This should allow us to not have documentation changes > >>>>> for code deployed before the final release.This should then allow us > >> to > >>>>> get rid of the site branch > >>>>> - Option 2: We keep the site branch, but we automate the current > >>>>> process. On every commit to master, if it is a change to the files in > >>>>> the site directory, we check if the change only touches certain > >>>>> whitelisted categories such as news and community. If so, we cherry > >> pick > >>>>> that into the site branch automatically using Github Actions and build > >>>>> and deploy the site. When a final release tag is pushed to the repo, > >> we > >>>>> use Github Actions to make the master and site branches equal and > >>>>> automatically build and deploy the site. > >>>>> > >>>>> This would negate the need to build and publish the site manually and > >>>>> simplify the process as we always only commit to master. As an added > >>>>> bonus, we if we keep the site branch, but automate the process, maybe > >> we > >>>>> can lock the site branch so that only CI can push to it. The downside > >> of > >>>>> course, is that we're relying on heuristics for the partial build, so > >>>>> there's some "magic" to it. > >>>>> > >>>>> Francis > >>>>> > >>>>> > >>>>> [1] https://issues.apache.org/jira/browse/CALCITE-3129 > >>>>> > >>>>> On 26/03/2022 8:58 am, Stamatis Zampetakis wrote: > >>>>>> Hello, > >>>>>> > >>>>>> Thanks for starting this discussion Liya. It is important to find > >> which > >>>>>> parts of the process are unclear and improve them if possible. > >>>>>> > >>>>>> The current procedure for updating the website remains unchanged and > >> it > >>>>> is > >>>>>> documented here: > >>>>>> > >>>>> > >>> > >> https://github.com/apache/calcite/blob/a6a1e2cef332893fd90286098869c56529e052c3/site/README.md > >>>>>> > >>>>>> If the procedure is not followed, which has happened a few times in > >> the > >>>>>> past, meaning that someone commits directly in site without > >> committing > >>> in > >>>>>> master then we will have commits in site that may get lost forever. > >>>>>> When we discover such commits we should port them to master. The > >>>>>> cherry-pick now goes in the opposite direction (from site to master). > >>>>>> This is usually discovered/done by the release manager and that's why > >>> we > >>>>>> have the respective instructions in the howto [1]. > >>>>>> > >>>>>> After a release we don't care much what happens because master and > >> site > >>>>>> should be equal. As Francis pointed out this is usually done with a > >>> force > >>>>>> push. > >>>>>> > >>>>>> Regarding Julian's question the commit hashes before the force pushes > >>>>> done > >>>>>> by Liya are the following (according to commits@calcite): > >>>>>> * master -> dcbc493bf699d961427952c5efc047b76d859096 > >>>>>> * site -> aa9dfc7dbc64c784040cf20ed168016ae3b9c2c5 > >>>>>> > >>>>>> Best, > >>>>>> Stamatis > >>>>>> > >>>>>> [1] > >>>>>> > >>>>> > >>> > >> https://github.com/apache/calcite/blob/a6a1e2cef332893fd90286098869c56529e052c3/site/_docs/howto.md?plain=1#L696 > >>>>>> > >>>>>> On Fri, Mar 25, 2022 at 7:36 PM Julian Hyde <[email protected]> > >> wrote: > >>>>>> > >>>>>>> Does anyone know (or could find out) the SHA of the master and site > >>>>>>> branches at the time that Fan attempted to move the site changes > >> over? > >>>>>>> If so, we could recreate the same environment, and figure out a set > >> of > >>>>>>> git commands that would have worked then and will work for the next > >>>>>>> release manager. This process is safe because we can do these > >>>>>>> experiments in a local git sandbox, without pushing to any remote. > >>>>>>> > >>>>>>> On Fri, Mar 25, 2022 at 6:09 AM Fan Liya <[email protected]> > >>> wrote: > >>>>>>>> > >>>>>>>> Hi Francis, > >>>>>>>> > >>>>>>>> Thanks for your feedback. > >>>>>>>> > >>>>>>>> It seems we should choose option 2. > >>>>>>>> In addition, it seems less risky to run "git push --force" commands > >>> in > >>>>>>>> the site branch. > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Liya Fan > >>>>>>>> > >>>>>>>> Francis Chuang <[email protected]> 于2022年3月25日周五 12:14写道： > >>>>>>>>> > >>>>>>>>> Hi Liya, > >>>>>>>>> > >>>>>>>>> Thanks for bringing this up. We have always done the following > >> when > >>>>>>>>> committing: > >>>>>>>>> 1. Always commit to master. > >>>>>>>>> 2. If we need to publish the change to the site now (for example, > >>> new > >>>>>>>>> committer or announcement), cherry-pick the change into the site > >>>>> branch > >>>>>>>>> and publish it. > >>>>>>>>> 3. After a release, make the site branch the same as master (git > >>> reset > >>>>>>>>> --hard master) and force push (git push --force origin site). > >>>>>>>>> > >>>>>>>>> Francis > >>>>>>>>> > >>>>>>>>> On 25/03/2022 3:03 pm, Fan Liya wrote: > >>>>>>>>>> Hi all, > >>>>>>>>>> > >>>>>>>>>> As part of the release process, we need to synchronize the master > >>> and > >>>>>>>>>> site branches (Please see > >>>>>>>>>> > >>>>>>> > >> https://calcite.apache.org/docs/howto.html#making-a-release-candidate > >>> ). > >>>>>>>>>> Usually, the site is behind the master branch by some commits. > >>>>>>>>>> If the existing commits in the site branch are in the same order > >> as > >>>>>>> in > >>>>>>>>>> the master branch, the task is easy: just switch to the site > >>> branch, > >>>>>>>>>> and run > >>>>>>>>>> > >>>>>>>>>> git rebase master > >>>>>>>>>> > >>>>>>>>>> However, if some commits are in different orders, it can be > >> tricky. > >>>>>>>>>> For example, the master branch may have the following commits (in > >>>>>>>>>> order): > >>>>>>>>>> > >>>>>>>>>> A, B, X1, X2, ... , Xn. > >>>>>>>>>> > >>>>>>>>>> and the site branch may have the following commits (in order): > >>>>>>>>>> > >>>>>>>>>> B, A, X1, X2. > >>>>>>>>>> > >>>>>>>>>> Basically we have two choices: > >>>>>>>>>> > >>>>>>>>>> 1. We can live with the out of order commits, because after > >>>>>>>>>> cherry-picking commits X3, X4, ... , Xn to the site branch, the > >>> file > >>>>>>>>>> contents will be consistent. > >>>>>>>>>> > >>>>>>>>>> The problem is that, since the two branches have diverged, we > >>> cannot > >>>>>>>>>> use the rebase command. Instead, we have to manually cherry-pick > >>>>>>>>>> commits individually, which requires large effort. In addition, > >> for > >>>>>>>>>> any subsequent release processes, we have to manually cherry-pick > >>>>>>> each > >>>>>>>>>> commit. > >>>>>>>>>> > >>>>>>>>>> 2. We need to make the commits order consistent, which will make > >> it > >>>>>>>>>> easy for subsequent releases. > >>>>>>>>>> However, the problem is that, to make the commits order > >> consistent, > >>>>>>>>>> some git force push command is unavoidable, which is risky to > >> some > >>>>>>>>>> extent. > >>>>>>>>>> > >>>>>>>>>> So what is the recommended way to do this? Thanks in advance for > >>>>>>> your feedback! > >>>>>>>>>> > >>>>>>>>>> Best, > >>>>>>>>>> Liya Fan > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > >
