Re: Removing documentation for old Beam versions

On Thu, Sep 27, 2018 at 7:25 PM Melissa Pashniak wrote: > > Ideally (IMO anyway), we would have versioned entire doc sets like most > Apache projects that I looked at do (Spark [1], Flink, Hadoop, etc.) with a > Latest + past releases, so users can read Beam docs appropriate to the > version they

Re: Removing documentation for old Beam versions

Ideally (IMO anyway), we would have versioned entire doc sets like most Apache projects that I looked at do (Spark [1], Flink, Hadoop, etc.) with a Latest + past releases, so users can read Beam docs appropriate to the version they are using. Would this run into the same situation as the javadoc/py

Re: Removing documentation for old Beam versions

Just to be clear, generated html for javadoc and pydoc will be put in apache/beam-site, but generated html for .md files will be put in apache/beam under the asf-site branch. On Wed, Sep 26, 2018 at 9:34 AM Thomas Weise wrote: > Looks like the is agreement that all sources should be in the main

Re: Removing documentation for old Beam versions

Looks like the is agreement that all sources should be in the main beam repository, the remaining discussion was where the generated content should be served from, specifically the generated docs. If the setup that Alan found allows us to keep using the beam-site repository for the generated stuff

Re: Removing documentation for old Beam versions

OK, thanks. That link was very helpful. Of the three options we must use, checking into git seems preferable than checking into svn let alone the CMS. Keeping the same repo means that it's harder to generate the docs for version X while head is checked out. I'm in favor of moving forward with this

Re: Removing documentation for old Beam versions

Yes. There are few options for publishing your ASF website, described here: https://www.apache.org/dev/project-site.html. We can publish from a Git repo, SVN, or a UI-based CMS interface. On Wed, Sep 26, 2018 at 9:45 AM Robert Bradshaw wrote: > I am also definitely in favor of a single repositor

Re: Removing documentation for old Beam versions

I am also definitely in favor of a single repository. Perhaps I'm just misunderstanding why the generated must be put in a git repository at all--is it because that's the easiest way to serve them? On Wed, Sep 26, 2018 at 3:39 PM Scott Wegner wrote: > Alan found the place where website publishin

Re: Removing documentation for old Beam versions

Alan found the place where website publishing is configured [1], which has examples of project sites being configured with more than one git root. This is great for us because it allows us to leave generated javadocs/pydocs in the beam-site repository and publish website markdown content from the m

Re: Removing documentation for old Beam versions

> We could add a new default branch (master?) and keep all the non-generated files (src/) there, and put generated files (content/) in the asf-site branch (like we already do). I'm strongly in favor of having sources in a single repository. We have significant process and infrastructure built up f

Re: Removing documentation for old Beam versions

Staying on beam-site SGTM. We could add a new default branch (master?) and keep all the non-generated files (src/) there, and put generated files (content/) in the asf-site branch (like we already do). That way there's no confusion as to which files you should update. (This is of course assuming we

Re: Removing documentation for old Beam versions

My thought was to leave the asf-site branch in the beam-site repository, add generated docs to that branch (until we have a better solution), and have only sources in the beam repo. Scott had filed https://issues.apache.org/jira/browse/BEAM-5459 - it would eliminate the need to place generated doc

Re: Removing documentation for old Beam versions

I believe that beam.apache.org is populated from the asf-site branch of the apache/beam-site repo. (gitpubsub: https://www.apache.org/dev/project-site.html#intro) If we move the markdown-based docs to apache/beam, leave generated javadoc and pydoc in apache/beam-site, and point gitpubsub to apache/

Re: Removing documentation for old Beam versions

Hi Scott, Thanks for bringing the discussion back here. I agree that we should separate the changes for hosting of generated java/pydocs from the rest of website automation so that we can make the switch and fix the contributor headache soon. But perhaps we can avoid adding 4m lines of generated

Re: Removing documentation for old Beam versions

Re-opening this thread as it came up today in the discussion for PR#6458 [1]. This PR is part of the work for Beam-Site Automation Reliability improvements; design doc here: https://s.apache.org/beam-site-automation The current plan is to keep generated javadoc/pydoc sources only on the asf-site b

Re: Removing documentation for old Beam versions

Hi Udi, Good to know you will continue this work. Let me know if you want to try the buildbot route (which does not require generated documentation to be checked into the repo). Happy to help with that. Thomas On Fri, Aug 24, 2018 at 11:36 AM Udi Meiri wrote: > I'm picking up the website migr

Re: Removing documentation for old Beam versions

Git is really efficient at things it can perform diffs on. Generated source code tends to be fine as long as it has reasonably short lines. It becomes an issue when you are checking in binaries, images, and compressed files (jars for example). On Fri, Aug 24, 2018 at 11:36 AM Udi Meiri wrote: >

Re: Removing documentation for old Beam versions

I'm picking up the website migration. The plan is to not include generated files in the master branch. However, I've been told that even putting generated files a separate branch could blow up the git repository for all (e.g. make git pulls a lot longer?). Not sure if this is a real issue or not.

Re: Removing documentation for old Beam versions

On Sun, Aug 5, 2018 at 5:28 AM Thomas Weise wrote: > > Yes, I think the separation of generated code will need to occur prior to > completing the merge and switching the web site to the main repo. > > There should be no reason to check generated documentation into either of the > repos/branches.

Re: Removing documentation for old Beam versions

Yes, I think the separation of generated code will need to occur prior to completing the merge and switching the web site to the main repo. There should be no reason to check generated documentation into either of the repos/branches. Please see as an example how this was solved in Flink, using the

Re: Removing documentation for old Beam versions

[image: pr-520.png] (trying that image again) On Thu, Aug 2, 2018 at 7:00 PM Udi Meiri wrote: > Alright, created https://github.com/apache/beam-site/pull/520 > [image: pr-520.png] > Reduces staging upload from 500M down to 270M, and halves the number of > files from ~22k to 11k. > > > > On Thu,

Re: Removing documentation for old Beam versions

Alright, created https://github.com/apache/beam-site/pull/520 [image: pr-520.png] Reduces staging upload from 500M down to 270M, and halves the number of files from ~22k to 11k. On Thu, Aug 2, 2018 at 6:58 PM Pablo Estrada wrote: > I believe tags will be necessarily because for anyone looking

Re: Removing documentation for old Beam versions

I believe tags will be necessarily because for anyone looking for old docs that have been removed, they will need to browse back in history, not just browse the tree of directories. -P. On Thu, Aug 2, 2018, 6:46 PM Mikhail Gryzykhin wrote: > Last time I talked with Scott I brought this idea in.

Re: Removing documentation for old Beam versions

Last time I talked with Scott I brought this idea in. I believe the plan was either to publish compiled site to website directly, or keep it in separate storage from apache/beam repo. One of the main reasons not to check in compiled version of website is that every developer will have to pull all

Re: Removing documentation for old Beam versions

Pablo, the docs are generated into versioned paths, e.g., https://beam.apache.org/documentation/sdks/javadoc/2.5.0/ so tags are not necessary? Also, once apache/beam-site is merged with apache/beam the release branch should have the relevant docs (although perhaps it's better to put them in a diffe

Re: Removing documentation for old Beam versions

+1 For removing old documentation. @Thomas: Migration work is in backlog and will be picked up in near time. --Mikhail Have feedback ? On Thu, Aug 2, 2018 at 12:54 PM Thomas Weise wrote: > +1 for removing pre 2.0 documentation (as well as the entries from > https:/

Re: Removing documentation for old Beam versions

+1 for removing pre 2.0 documentation (as well as the entries from https://beam.apache.org/get-started/downloads/) Isn't it part of the beam-site changes that we will no longer check in generated documentation into the repository? Those can be generated and deployed independently (when a commit to

Re: Removing documentation for old Beam versions

Is it worth adding a tag / branch to the repositories every time we make a release, so that people are able to dive in and find the docs? Best -P. On Thu, Aug 2, 2018 at 12:09 PM Ahmet Altay wrote: > I would guess that users are still using some of these old releases. It is > unclear from Beam w

Re: Removing documentation for old Beam versions

I would guess that users are still using some of these old releases. It is unclear from Beam website which releases are still supported or not. It probably makes sense to drop documentation for releases < 2.0. (I would suggest keeping docs for 2.0). For the future I can work on updating the Beam we

Re: Removing documentation for old Beam versions

The older docs are not directly linked to and are in Github commit history. If there are no objections I'm going to delete javadocs and pydocs for releases older than 1 year, meaning 2.0.0 and older (going by the dates here ). On Thu, Aug 2, 2018 at

Re: Removing documentation for old Beam versions

The older docs should be recorded in the commit history of the website repository, right? If they're not currently used in the website and they're in the commit history then I don't see a reason to save them. On Tue, Jul 31, 2018 at 1:51 PM Udi Meiri wrote: > Hi all, > I'm writing a PR for apach

Removing documentation for old Beam versions

Hi all, I'm writing a PR for apache/beam-site and beam_PreCommit_Website_Stage is timing out after 100 minutes, because it's trying to deletes 22k files and then copy 22k files (warning large file ). It seems that we cou