Re: Bundle Documentation: Readme in Bundle Repo vs MD in Site Repo

[Bertrand Delacretaz]

On Wed, Oct 4, 2017 at 12:03 PM, Oliver Lietz  wrote:
> ...I prefer having a good
> README or module "site" and only documenting high-level concepts or general
> stuff in Sling's site (and wiki)...

+1

-Bertrand

Re: Bundle Documentation: Readme in Bundle Repo vs MD in Site Repo

[Oliver Lietz]

On Tuesday 03 October 2017 15:50:17 Konrad Windszus wrote:
> I think we also once discussed on the mailing list where to put bundle
> documentation: either in the readme or in the site itself. Unfortunately I
> couldn't find the according thread. Please help me with the link if you
> have it handy.
> 
> Right now the information is a bit distributed and most of the readme's only
> contained a minimally adjusted template and most of the information from a
> consumer's perspective can be found MD pages below
> https://sling.apache.org/documentation/bundles.html. How do we want to
> proceed in the future?
> 
> # Put the bundle-specific information rather in the readme to make it easier
> to adjust together with the code (in one common PR) # Keep maintaining a
> dedicated MD page in the site repository (to make it easier to cross-link
> to other pages, probably also better from SEO perspective)
> 
> And the other question is how do we properly cross-link from one to the
> other. At least for option 1 I would like to put a link in
> https://sling.apache.org/documentation/bundles.html to the individual
> readme files. Since right now we are updating all readme files I think we
> should include this aspect as well.

Besides moving content from READMEs next to reactor/builder POMs I have no 
plans on spending more time with them (the goal was to have at least a minimal 
README with a uniform header in each module before splitting up into ~300 Git 
repos).

>From my experience documentation in wikis or other external documents outdates 
much faster than documentation close to code. Therefore I prefer having a good 
README or module "site" and only documenting high-level concepts or general 
stuff in Sling's site (and wiki).

Regards,
O.

> Once we have an agreement we should document that in
> https://sling.apache.org/documentation.html.
> 
> Thanks for your opinions
> Konrad

Re: Bundle Documentation: Readme in Bundle Repo vs MD in Site Repo

[Konrad Windszus]

Thanks for the pointers. I created 
https://issues.apache.org/jira/browse/SLING-7176 to track this.
Feel free to extend it with your own ideas.

> On 4. Oct 2017, at 11:49, Bertrand Delacretaz  wrote:
> 
> Hi,
> 
> On Wed, Oct 4, 2017 at 9:57 AM, Robert Munteanu  wrote:
>> ...The 'canonical' way of doing this is by accessing https://api.github.co
>> m/orgs/${github_org}/repos and filtering based on the repo name...
> 
> Note that this would require handling multiple pages, it looks like
> the max number of results is 200:
> 
> $ curl -s "https://api.github.com/orgs/apache/repos?page=1_page=200;
> | grep html_url | wc -l
> 200
> $ curl -s "https://api.github.com/orgs/apache/repos?page=1_page=210;
> | grep html_url | wc -l
> 200
> $ curl -s "https://api.github.com/orgs/apache/repos?page=2_page=210;
> | grep html_url | wc -l
> 200
> 
> But otherwise grabbing that is easy in our Groovy templates, something like:
> 
> URL url = new 
> URL("https://api.github.com/orgs/apache/repos?page=1_page=100;)
> def json = new groovy.json.JsonSlurper().parseText(url.text)
> json.each ( {
>it ->
>if(it.full_name.contains("struts")) {
>println it.html_url
>}
> })
> 
> -Bertrand

Re: Bundle Documentation: Readme in Bundle Repo vs MD in Site Repo

[Robert Munteanu]

On Wed, 2017-10-04 at 11:49 +0200, Bertrand Delacretaz wrote:
> Hi,
> 
> On Wed, Oct 4, 2017 at 9:57 AM, Robert Munteanu 
> wrote:
> > ...The 'canonical' way of doing this is by accessing https://api.gi
> > thub.co
> > m/orgs/${github_org}/repos and filtering based on the repo name...
> 
> Note that this would require handling multiple pages, it looks like
> the max number of results is 200:
> 
> $ curl -s "https://api.github.com/orgs/apache/repos?page=1_page=2
> 00"
> > grep html_url | wc -l
> 
>  200
> $ curl -s "https://api.github.com/orgs/apache/repos?page=1_page=2
> 10"
> > grep html_url | wc -l
> 
>  200
> $ curl -s "https://api.github.com/orgs/apache/repos?page=2_page=2
> 10"
> > grep html_url | wc -l
> 
>  200
> 
> But otherwise grabbing that is easy in our Groovy templates,
> something like:
> 
> URL url = new URL("https://api.github.com/orgs/apache/repos?page=1
> r_page=100")
> def json = new groovy.json.JsonSlurper().parseText(url.text)
> json.each ( {
> it ->
> if(it.full_name.contains("struts")) {
> println it.html_url
> }
> })

I'll just paste here the chunks script I've used so far. I won't link
to it since it's in the 'not-sling' org which will go away next week.

github_org="not-sling"
project_prefix="sling-"
curl_args="$1"

page_count=$(curl ${curl_args} -sI https://api.github.com/orgs/${github
_org}/repos | grep 'Link: ' | tr ',' '\n' | grep 'rel="last"' |
sed  's/.*page=\([0-9][0-9]*\).*/\1/')

for page in $(seq 1 ${page_count}); do
for repo in $(curl ${curl_args} -s https://api.github.com/orgs/${gi
thub_org}/repos?page=${page} \
   | jq ".[] |  .name | match (\"${project_prefix}.*\") | .string"
| tr -d '"' | sort); do
# your processing here
done
done

We can consider either creating a shared script for it or have a single
job which periodically regenerates a JSON file with all the projects.

Robert

Re: Bundle Documentation: Readme in Bundle Repo vs MD in Site Repo

[Bertrand Delacretaz]

Hi,

On Wed, Oct 4, 2017 at 9:57 AM, Robert Munteanu  wrote:
> ...The 'canonical' way of doing this is by accessing https://api.github.co
> m/orgs/${github_org}/repos and filtering based on the repo name...

Note that this would require handling multiple pages, it looks like
the max number of results is 200:

$ curl -s "https://api.github.com/orgs/apache/repos?page=1_page=200;
| grep html_url | wc -l
 200
$ curl -s "https://api.github.com/orgs/apache/repos?page=1_page=210;
| grep html_url | wc -l
 200
$ curl -s "https://api.github.com/orgs/apache/repos?page=2_page=210;
| grep html_url | wc -l
 200

But otherwise grabbing that is easy in our Groovy templates, something like:

URL url = new 
URL("https://api.github.com/orgs/apache/repos?page=1_page=100;)
def json = new groovy.json.JsonSlurper().parseText(url.text)
json.each ( {
it ->
if(it.full_name.contains("struts")) {
println it.html_url
}
})

-Bertrand

Re: Bundle Documentation: Readme in Bundle Repo vs MD in Site Repo

[Robert Munteanu]

On Wed, 2017-10-04 at 10:01 +0200, Konrad Windszus wrote:
> > On 4. Oct 2017, at 09:57, Robert Munteanu <romb...@apache.org>
> > wrote:
> > 
> > On Tue, 2017-10-03 at 17:03 +0200, Konrad Windszus wrote:
> > > > On 3. Oct 2017, at 16:35, Bertrand Delacretaz <bdelacretaz@apac
> > > > he.o
> > > > rg> wrote:
> > > > 
> > > > Hi,
> > > > 
> > > > On Tue, Oct 3, 2017 at 3:50 PM, Konrad Windszus <konrad_w@gmx.d
> > > > e>
> > > > wrote:
> > > > > ...I think we also once discussed on the mailing list where
> > > > > to
> > > > > put bundle documentation: either in the readme or in the site
> > > > > itself...
> > > > 
> > > > Now that we have complete control over the website we might
> > > > generate
> > > > http://sling.apache.org/documentation/bundles.html (or part of
> > > > it)
> > > > using the GitHub API, or the module list that we use for
> > > > Jenkins,
> > > > and
> > > > mostly use the module's README.md from now on.
> > > 
> > > That sounds really good.
> > > @Robert: Would it be possible to maintain the module list as
> > > separate
> > > JSON file (currently contained in
> > > https://svn.apache.org/repos/asf/sling/trunk/tooling/jenkins/crea
> > > te_j
> > > obs.groovy).
> > 
> > My plan is to access the repositories using the Sling API rather
> > than
> > define a big list which can get out of date.
> 
> I guess you mean GitHub API, right :-)?

Right :-)

Re: Bundle Documentation: Readme in Bundle Repo vs MD in Site Repo

[Konrad Windszus]

> On 4. Oct 2017, at 09:57, Robert Munteanu <romb...@apache.org> wrote:
> 
> On Tue, 2017-10-03 at 17:03 +0200, Konrad Windszus wrote:
>>> On 3. Oct 2017, at 16:35, Bertrand Delacretaz <bdelacretaz@apache.o
>>> rg> wrote:
>>> 
>>> Hi,
>>> 
>>> On Tue, Oct 3, 2017 at 3:50 PM, Konrad Windszus <konra...@gmx.de>
>>> wrote:
>>>> ...I think we also once discussed on the mailing list where to
>>>> put bundle documentation: either in the readme or in the site
>>>> itself...
>>> 
>>> Now that we have complete control over the website we might
>>> generate
>>> http://sling.apache.org/documentation/bundles.html (or part of it)
>>> using the GitHub API, or the module list that we use for Jenkins,
>>> and
>>> mostly use the module's README.md from now on.
>> 
>> That sounds really good.
>> @Robert: Would it be possible to maintain the module list as separate
>> JSON file (currently contained in
>> https://svn.apache.org/repos/asf/sling/trunk/tooling/jenkins/create_j
>> obs.groovy).
> 
> My plan is to access the repositories using the Sling API rather than
> define a big list which can get out of date.
I guess you mean GitHub API, right :-)?
> 
> The 'canonical' way of doing this is by accessing https://api.github.co
> m/orgs/${github_org}/repos and filtering based on the repo name.
> 
>> 
>> Then we could easily use it for this purpose as well.
>> The only question would be, when and how to trigger the update.
>> 
>> Also we would have two different possibilities here:
>> 1. Just link towards the readme.md (we must link to Github though, as
>> Apache's Gitbox doesn't seem to be able to render the HTML from the
>> MD, compare with https://gitbox.apache.org/repos/asf?p=nutch.git;a=bl
>> ob;f=README.md;h=3aed205beb13d5b76c257d192d6c23d94669be35;hb=HEAD)
>> 2. Generate the HTML page within the Sling site for each found
>> readme.
>> 
>> I would rather be in favour of option 2 though, as otherwise the
>> readme link would only point towards Github, but of course then the
>> updating issue is even more interesting...
> 
> Option 2 sounds good to me as well. I am not sure though, as Bertrand
> already mentioned, that we can rely on the README.md format rendering
> 100% as expected in a different context.

I am pretty sure that there are slight differences on how the MD files is 
rendered with JBake 2.5.1 and with GitHub, so most probably some documentation 
will not work on either GitHub or Sling Site.
> 
> What we could do is:
> 
> - always keep the documentation in sling.apache.org
> - add a deep link from the README.md to the specific documentation page
> on sling.apache.org .
> 
> Thanks,
> 
> Robert

Re: Bundle Documentation: Readme in Bundle Repo vs MD in Site Repo

[Robert Munteanu]

On Tue, 2017-10-03 at 17:03 +0200, Konrad Windszus wrote:
> > On 3. Oct 2017, at 16:35, Bertrand Delacretaz <bdelacretaz@apache.o
> > rg> wrote:
> > 
> > Hi,
> > 
> > On Tue, Oct 3, 2017 at 3:50 PM, Konrad Windszus <konra...@gmx.de>
> > wrote:
> > > ...I think we also once discussed on the mailing list where to
> > > put bundle documentation: either in the readme or in the site
> > > itself...
> > 
> > Now that we have complete control over the website we might
> > generate
> > http://sling.apache.org/documentation/bundles.html (or part of it)
> > using the GitHub API, or the module list that we use for Jenkins,
> > and
> > mostly use the module's README.md from now on.
> 
> That sounds really good.
> @Robert: Would it be possible to maintain the module list as separate
> JSON file (currently contained in
> https://svn.apache.org/repos/asf/sling/trunk/tooling/jenkins/create_j
> obs.groovy).

My plan is to access the repositories using the Sling API rather than
define a big list which can get out of date.

The 'canonical' way of doing this is by accessing https://api.github.co
m/orgs/${github_org}/repos and filtering based on the repo name.

> 
> Then we could easily use it for this purpose as well.
> The only question would be, when and how to trigger the update.
> 
> Also we would have two different possibilities here:
> 1. Just link towards the readme.md (we must link to Github though, as
> Apache's Gitbox doesn't seem to be able to render the HTML from the
> MD, compare with https://gitbox.apache.org/repos/asf?p=nutch.git;a=bl
> ob;f=README.md;h=3aed205beb13d5b76c257d192d6c23d94669be35;hb=HEAD)
> 2. Generate the HTML page within the Sling site for each found
> readme.
> 
> I would rather be in favour of option 2 though, as otherwise the
> readme link would only point towards Github, but of course then the
> updating issue is even more interesting...

Option 2 sounds good to me as well. I am not sure though, as Bertrand
already mentioned, that we can rely on the README.md format rendering
100% as expected in a different context.

What we could do is:

- always keep the documentation in sling.apache.org
- add a deep link from the README.md to the specific documentation page
on sling.apache.org .

Thanks,

Robert

Re: Bundle Documentation: Readme in Bundle Repo vs MD in Site Repo

[Bertrand Delacretaz]

Hi,

On Tue, Oct 3, 2017 at 5:03 PM, Konrad Windszus  wrote:
> ...The only question would be, when and how to trigger the update...

We touch the site regularly, so the list of modules would be kept up
to date frequently - and if needed it's easy to regenerate the website
when adding a new module.

> ...Also we would have two different possibilities here:
> 1. Just link towards the readme.md (we must link to Github...
> 2. Generate the HTML page within the Sling site for each found readme...

I was thinking of the first option, I suppose the second one would be
problematic if README.s including inline images etc.

But we might including an excerpt of the READMEs on the links page, or
at least their title.

-Bertrand

Re: Bundle Documentation: Readme in Bundle Repo vs MD in Site Repo

[Konrad Windszus]

> On 3. Oct 2017, at 16:35, Bertrand Delacretaz <bdelacre...@apache.org> wrote:
> 
> Hi,
> 
> On Tue, Oct 3, 2017 at 3:50 PM, Konrad Windszus <konra...@gmx.de> wrote:
>> ...I think we also once discussed on the mailing list where to put bundle 
>> documentation: either in the readme or in the site itself...
> 
> Now that we have complete control over the website we might generate
> http://sling.apache.org/documentation/bundles.html (or part of it)
> using the GitHub API, or the module list that we use for Jenkins, and
> mostly use the module's README.md from now on.

That sounds really good.
@Robert: Would it be possible to maintain the module list as separate JSON file 
(currently contained in 
https://svn.apache.org/repos/asf/sling/trunk/tooling/jenkins/create_jobs.groovy).

Then we could easily use it for this purpose as well.
The only question would be, when and how to trigger the update.

Also we would have two different possibilities here:
1. Just link towards the readme.md (we must link to Github though, as Apache's 
Gitbox doesn't seem to be able to render the HTML from the MD, compare with 
https://gitbox.apache.org/repos/asf?p=nutch.git;a=blob;f=README.md;h=3aed205beb13d5b76c257d192d6c23d94669be35;hb=HEAD)
2. Generate the HTML page within the Sling site for each found readme.

I would rather be in favour of option 2 though, as otherwise the readme link 
would only point towards Github, but of course then the updating issue is even 
more interesting...

Konrad

> 
> Patches welcome ;-)
> 
> -Bertrand

Re: Bundle Documentation: Readme in Bundle Repo vs MD in Site Repo

[Robert Munteanu]

On Tue, 2017-10-03 at 15:50 +0200, Konrad Windszus wrote:
> I think we also once discussed on the mailing list where to put
> bundle documentation: either in the readme or in the site itself.
> Unfortunately I couldn't find the according thread. Please help me
> with the link if you have it handy.

https://lists.apache.org/thread.html/b7523b72d99b68f96313bbe0622fda77ce
afc26351caa790a55b67dc@%3Cdev.sling.apache.org%3E

Robert

Re: Bundle Documentation: Readme in Bundle Repo vs MD in Site Repo

[Bertrand Delacretaz]

Hi,

On Tue, Oct 3, 2017 at 3:50 PM, Konrad Windszus <konra...@gmx.de> wrote:
> ...I think we also once discussed on the mailing list where to put bundle 
> documentation: either in the readme or in the site itself...

Now that we have complete control over the website we might generate
http://sling.apache.org/documentation/bundles.html (or part of it)
using the GitHub API, or the module list that we use for Jenkins, and
mostly use the module's README.md from now on.

Patches welcome ;-)

-Bertrand

Bundle Documentation: Readme in Bundle Repo vs MD in Site Repo

[Konrad Windszus]

I think we also once discussed on the mailing list where to put bundle 
documentation: either in the readme or in the site itself. Unfortunately I 
couldn't find the according thread. Please help me with the link if you have it 
handy.

Right now the information is a bit distributed and most of the readme's only 
contained a minimally adjusted template and most of the information from a 
consumer's perspective can be found MD pages below 
https://sling.apache.org/documentation/bundles.html. How do we want to proceed 
in the future?

# Put the bundle-specific information rather in the readme to make it easier to 
adjust together with the code (in one common PR)
# Keep maintaining a dedicated MD page in the site repository (to make it 
easier to cross-link to other pages, probably also better from SEO perspective)

And the other question is how do we properly cross-link from one to the other. 
At least for option 1 I would like to put a link in 
https://sling.apache.org/documentation/bundles.html to the individual readme 
files.
Since right now we are updating all readme files I think we should include this 
aspect as well.

Once we have an agreement we should document that in 
https://sling.apache.org/documentation.html.

Thanks for your opinions
Konrad

Bundle Documentation

[Daniel Klco]

All,

I was wondering if there was any interest in producing a documentation site
for the various Sling Bundles?  I noticed there was a blurb about it on
this page:
http://sling.apache.org/documentation.html#the-bundle-documentation

And it sounded like an interesting idea to me.  I went ahead and tossed
together a Maven Skin which produces pages quite similar to the current
Sling site, you can see the code here:
https://svn.apache.org/repos/asf/sling/whiteboard/dklco/bundle-skin/

And a sample site produced here:
http://sling-proxy.danklco.com/

I think using the SCM Publish Plugin (
http://maven.apache.org/plugins/maven-scm-publish-plugin/usage.html) and
some path naming convention we could probably set up all of the bundles to
publish directly into the Apache CMS Staging with one command.

WDYT?

-Dan