Thank you Chia-Ping. Conversion is mostly automated ( https://github.com/hvishwanath/ak2md ) and as such the prototype <https://kafka-site-md-711970345036.us-central1.run.app/> supports all previous versions. However, not having to support all previous versions will definitely be simpler.
Hugo supports live-reload. The docsy theme has dependencies on golang and a couple of nodeJS libraries (autoprefixer, postcss etc.,) and as such it will be simpler to use docker based workflows to avoid having to install any dependencies locally for development (ref: https://github.com/hvishwanath/kafka-site-md/blob/main/README.md#local-development-server ). We can set up CI pipelines to build and push production images serving the site with nginx. Regards, Harish On Sun, Mar 9, 2025 at 9:08 PM Chia-Ping Tsai <chia7...@gmail.com> wrote: > hi Harish > > I have to say, this KIP is one of the best ideas I've seen recently, and I > agree with Matthias that we don't need to translate any older versions. I > truly hope this KIP can be included in 4.0. > > I have a quick question: what is the recommended procedure for testing the > site locally after migrating to Markdown? Previously, we used a container > to host the site locally for previewing changes. Could you please provide > guidance on the updated workflow? > > Best, > Chia-Ping > > Harish Vishwanath <harish.shas...@gmail.com> 於 2025年3月10日 週一 上午11:53寫道： > > > Hello all > > > > Sorry for my delayed response. Thanks for your feedback and comments. > > Please see below: > > > > > Do you plan to have a development version of the website online in > > parallel to the old HTML version? > > Yes. The current prototype which supports all versions until AK 3.9 is > > running at https://kafka-site-md-711970345036.us-central1.run.app/, the > > prototype > > < > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown#KIP1133:AKDocumentationandWebsiteinMarkdown-Prototype > > > > > section of the KIP has more details. We should be able to host this in > ASF > > infra. Further, as described in the KIP > > > > >I understand that locally with the markdown files and an markdown > editor, > > we see the rough structure of the docs (which is already an improvement), > > but we will not see the end product, because for that we again need a web > > server. > > Hugo supports live reload making local development easier (ref: > > https://github.com/hvishwanath/kafka-site-md/blob/main/Makefile#L25) > > > > > > [MM1] Yes., I have ensured that all existing auto generated > documentations > > (html files under javadoc, generated folders) are available at the same > > relative location as they are in the current website. You can see this in > > the prototype. > > For example: > > > > > https://kafka.apache.org/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > is > > available at > > > > > https://kafka-site-md-711970345036.us-central1.run.app/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > < > > > https://kafka-site-md-711970345036.us-central1.run.app/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > > > Furthermore, for production, we can serve this with a full blown web > server > > such as nginx allowing us to set up any required redirects. > > > > [MM2] I recommend that we leave any auto generated documentation to > > continue to be in HTML format. We can serve them as static assets (ref: > > https://github.com/hvishwanath/kafka-site-md/tree/main/static) > > > > >I am also in favor of having easy access to `trunk / SNAPSHOT` version > of > > the docs if possible. > > Yes, this is definitely possible. > > > > > > Regards, > > Harish > > > > > > On Sat, Mar 1, 2025 at 3:01 PM Matthias J. Sax <mj...@apache.org> wrote: > > > > > Hi, > > > > > > I am totally in favor to make this happen, and I was disappointed each > > > time in the past, when it was proposed but we did not do it. > > > > > > > > > > > > Given that it is a huge effort, I am wondering if it would be > sufficient > > > to just to this change for 4.0 release though, and not translate any > > > older versions at all? > > > > > > > > > For including the docs in the release, I am actually wondering what the > > > value is? Does anybody actually use these artifact? Of course, we need > > > to publish JavaDocs, but the content of the web page seems to have very > > > low value to the community? -- I did also check ASF guidelines about it > > > and the webpage [1] say: > > > > > > > This material may include documentation concerning the release [...] > > > > > > So it is optional, but not required to publish docs artifacts from my > > > understanding. I have no objections to keep publishing the web page > > > docs, just saying, if the effort is too large and the value to low, we > > > could consider just dropping it. > > > > > > > > > I am also in favor of having easy access to `trunk / SNAPSHOT` version > > > of the docs if possible. Some other ASF project do this already, for > > > example Flink [2]. It's much easier to work with, and allows to spot > > > issues during development. While it does not help on a PR directly, it > > > is still a good way to verify a PR after it was merged, so any errors > > > can be detected and corrected more easily. > > > > > > > > > Also agree to the other raised points about generated content and > > > permalinks. For permalinks, if effort is reasonable, I would rather try > > > to redirect as many as possible. > > > > > > > > > > > > -Matthias > > > > > > > > > > > > > > > [1] > > > > > > https://www.apache.org/legal/release-policy.html#distribute-other-artifacts > > > [2] https://nightlies.apache.org/flink/flink-docs-master/ > > > > > > > > > On 2/25/25 8:43 AM, David Arthur wrote: > > > > MM1: I suspect Hugo has some way of doing url redirection, but at the > > > very > > > > least we can add redirects in our htaccess file > > > > https://github.com/apache/kafka-site/blob/asf-site/.htaccess > > > > > > > > Excluding the "front matter" (intro, quickstart, powered by, etc) I > > think > > > > the main links we need to worry about are the configuration docs > pages. > > > In > > > > the demo, it looks like we are using the same anchors for specific > > config > > > > sections. > > > > > > > > E.g., > > > > > > > > > > https://kafka-site-md-711970345036.us-central1.run.app/39/configuration/configuration/#topicconfigs_compression.gzip.level > > > > is the same content as > > > > > > > > > > https://kafka.apache.org/documentation/#topicconfigs_compression.zstd.level > > > > > > > > So maybe it shouldn't be too hard? > > > > > > > > Personally, I think the only links we should consider to be > > "permalinks" > > > > are the front matter and the documentation pages with the version in > > the > > > > URL. The unversioned documentation pages (like > > > > > > > > > > https://kafka.apache.org/documentation/#topicconfigs_compression.zstd.level > > > ) > > > > can change from release to release, so they are not expected to be > > > > permanent. > > > > > > > > Do we have any data on inbound links to the site? Maybe from Google > > > > Analytics or similar? > > > > > > > > -David A > > > > > > > > > > > > > > > > On Mon, Feb 24, 2025 at 10:17 AM Mickael Maison < > > > mickael.mai...@gmail.com> > > > > wrote: > > > > > > > >> Hi Harish, > > > >> > > > >> This is definitively something we want to do! > > > >> > > > >> MM1. One aspect to keep in mind in terms of compatibility is whether > > > >> existing links will still work. Thousands of online resources link > to > > > >> the Apache Kafka website. Is it possible to keep all links working? > > > >> > > > >> MM2. A bunch of sections are generated. To do so we have a number of > > > >> classes in the public API (for example 0, 1) with toHtml() or main() > > > >> methods that output HTML. Should we update these to output Markdown > > > >> directly instead? > > > >> > > > >> Converting the website has been discussed several times, I'd suggest > > > >> checking at least https://issues.apache.org/jira/browse/KAFKA-2967 > to > > > >> see what was previously said. > > > >> > > > >> 0: > > > >> > > > > > > https://kafka.apache.org/39/javadoc/org/apache/kafka/common/config/ConfigDef.html#toHtml() > > > >> 1: > > > >> > > > > > > https://kafka.apache.org/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > >> > > > >> Thanks, > > > >> Mickael > > > >> > > > >> On Mon, Feb 24, 2025 at 3:33 PM Bruno Cadonna <cado...@apache.org> > > > wrote: > > > >>> > > > >>> Hi Harish, > > > >>> > > > >>> Thanks for the KIP! Great initiative! > > > >>> > > > >>> Do you plan to have a development version of the website online in > > > >>> parallel to the old HTML version? > > > >>> > > > >>> I understand that locally with the markdown files and an markdown > > > >>> editor, we see the rough structure of the docs (which is already an > > > >>> improvement), but we will not see the end product, because for that > > we > > > >>> again need a web server. > > > >>> > > > >>> I assume that after you migrated all docs to markdown, we want to > see > > > >>> the complete website, review the content, and make some > refinements. > > > >>> > > > >>> Best, > > > >>> Bruno > > > >>> > > > >>> On 23.02.25 07:52, Harish Vishwanath wrote: > > > >>>> Thank you David. Appreciate your feedback. My comments below: > > > >>>> > > > >>>> DA1) I think this is a very good suggestion. While building the > > > >> prototype, > > > >>>> I did spend additional effort for older versions of the doc since > > > >> there are > > > >>>> fairly significant differences in the structure and format of the > > > >> documents > > > >>>> as compared to 3.x versions. I have written the automation which > can > > > do > > > >>>> this, but I do think hosting the older versions (<3.x) in its > > current > > > >>>> format and moving the rest to markdown will reduce the overall > > > >>>> testing/validation effort. That said, if we foresee making > > relatively > > > >>>> frequent updates to older versions, it might be worth converting > > > >> everything > > > >>>> to markdown. > > > >>>> > > > >>>> DA2) My hunch is that this can be quite involved. Rendering a > usable > > > >> HTML > > > >>>> version requires all the supporting artifacts (css, js, images, > hugo > > > >>>> shortcodes, partials etc.,) which will only be present in the > > website > > > >>>> documentation. Further, the hyper links themselves will not work > > > >> without > > > >>>> a webserver. To serve a directory structure based HTML, it will > > > >> require us > > > >>>> to have different templates for these docs where dependencies on > the > > > >>>> artifacts I mentioned above will need to be removed, links will > need > > > >> to be > > > >>>> converted to use relative directory locations etc., > > > >>>> > > > >>>> DA3) This is definitely possible. Using tools such as `pandoc` we > > > >> should be > > > >>>> able to render plain text versions of markdown files. For ex, the > > > below > > > >>>> would create a plain text version of all docs for 39/ branch., > which > > > >> can be > > > >>>> packaged with the binary distribution. > > > >>>> > > > >>>> find docs/ -name "*.md" -type f | while read -r file; do > > > >>>> dir=$(dirname "$file") > > > >>>> filename=$(basename "$file" .md) > > > >>>> mkdir -p "39-ascii/$dir" > > > >>>> pandoc -f markdown -t plain "$file" -o > > > >> "39-ascii/$dir/$filename.txt" > > > >>>> done > > > >>>> > > > >>>> DA4) Makes sense. I updated the wiki to reflect this. > > > >>>> > > > >>>> > > > >>>> Regards, > > > >>>> Harish > > > >>>> > > > >>>> > > > >>>> On Fri, Feb 21, 2025 at 8:56 AM David Arthur <mum...@gmail.com> > > > wrote: > > > >>>> > > > >>>>> Thanks for the KIP, Harish! > > > >>>>> > > > >>>>> I am +100 just on the title alone :) > > > >>>>> > > > >>>>> The motivations in the KIP are sound. All of these are very real > > > >> annoyances > > > >>>>> when it comes to maintaining our docs. I suspect we have not done > > > >> anything > > > >>>>> to improve them since the system we have worked well enough in > the > > > >> past. A > > > >>>>> better way to say that is: it hasn't been bad enough to motivate > us > > > to > > > >>>>> change it. > > > >>>>> > > > >>>>> With efforts like this, there is always some question about which > > > >> tool or > > > >>>>> method we should use. I like that your proposal includes these > > > >> specifics as > > > >>>>> well as a complete and functional demo. > > > >>>>> > > > >>>>> Few questions: > > > >>>>> > > > >>>>> DA1) Do we really need to convert all of our docs to the new > thing? > > > It > > > >>>>> might be reasonable to draw a line somewhere (maybe 3.0?) and > have > > > >> docs > > > >>>>> prior to that be hosted elsewhere (kafka.apache.org/old or > > > >> something). > > > >>>>> This > > > >>>>> may or may not be more work than just converting everything. > WDYT? > > > >>>>> > > > >>>>> DA2) For our packaged docs (those that go in the binaries we > > > >> distribute), > > > >>>>> do you think it's possible to render HTML that can be viewed > > without > > > a > > > >>>>> webserver? Currently our packaged docs are kind of useless IMO. > It > > > >> would be > > > >>>>> better if they could be viewed without any extra effort by an > > > >> operator. > > > >>>>> > > > >>>>> DA3) As a possible alternative to DA2, could we render the > markdown > > > >> docs as > > > >>>>> ascii text files? This is arguably better for operators since > they > > > >> don't > > > >>>>> always have a GUI or web browser handy. > > > >>>>> > > > >>>>> DA4) Just a nitpick, could you reduce the "Sample directory > layout" > > > >> down to > > > >>>>> one or two examples? It's difficult to see the versioned vs > > > >> non-versioned > > > >>>>> docs in the current block > > > >>>>> > > > >>>>> Thanks! > > > >>>>> David A > > > >>>>> > > > >>>>> > > > >>>>> On Sun, Feb 16, 2025 at 2:10 PM Harish Vishwanath < > > > >>>>> harish.shas...@gmail.com> > > > >>>>> wrote: > > > >>>>> > > > >>>>>> Hello, > > > >>>>>> > > > >>>>>> Following up from my previous email regarding moving AK > > > >> documentation to > > > >>>>>> markdown, I wrote KIP-1133 > > > >>>>>> < > > > >>>>>> > > > >>>>> > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown > > > >>>>>>> > > > >>>>>> detailing > > > >>>>>> the proposal and would like to start discussions. Please take a > > > look. > > > >>>>>> > > > >>>>>> Regards, > > > >>>>>> Harish > > > >>>>>> > > > >>>>> > > > >>>>> > > > >>>>> -- > > > >>>>> David Arthur > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > > > > > > > > > > > >
