hi Harish I have to say, this KIP is one of the best ideas I've seen recently, and I agree with Matthias that we don't need to translate any older versions. I truly hope this KIP can be included in 4.0.
I have a quick question: what is the recommended procedure for testing the site locally after migrating to Markdown? Previously, we used a container to host the site locally for previewing changes. Could you please provide guidance on the updated workflow? Best, Chia-Ping Harish Vishwanath <harish.shas...@gmail.com> 於 2025年3月10日 週一 上午11:53寫道： > Hello all > > Sorry for my delayed response. Thanks for your feedback and comments. > Please see below: > > > Do you plan to have a development version of the website online in > parallel to the old HTML version? > Yes. The current prototype which supports all versions until AK 3.9 is > running at https://kafka-site-md-711970345036.us-central1.run.app/, the > prototype > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown#KIP1133:AKDocumentationandWebsiteinMarkdown-Prototype > > > section of the KIP has more details. We should be able to host this in ASF > infra. Further, as described in the KIP > > >I understand that locally with the markdown files and an markdown editor, > we see the rough structure of the docs (which is already an improvement), > but we will not see the end product, because for that we again need a web > server. > Hugo supports live reload making local development easier (ref: > https://github.com/hvishwanath/kafka-site-md/blob/main/Makefile#L25) > > > [MM1] Yes., I have ensured that all existing auto generated documentations > (html files under javadoc, generated folders) are available at the same > relative location as they are in the current website. You can see this in > the prototype. > For example: > > https://kafka.apache.org/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > is > available at > > https://kafka-site-md-711970345036.us-central1.run.app/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > < > https://kafka-site-md-711970345036.us-central1.run.app/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > Furthermore, for production, we can serve this with a full blown web server > such as nginx allowing us to set up any required redirects. > > [MM2] I recommend that we leave any auto generated documentation to > continue to be in HTML format. We can serve them as static assets (ref: > https://github.com/hvishwanath/kafka-site-md/tree/main/static) > > >I am also in favor of having easy access to `trunk / SNAPSHOT` version of > the docs if possible. > Yes, this is definitely possible. > > > Regards, > Harish > > > On Sat, Mar 1, 2025 at 3:01 PM Matthias J. Sax <mj...@apache.org> wrote: > > > Hi, > > > > I am totally in favor to make this happen, and I was disappointed each > > time in the past, when it was proposed but we did not do it. > > > > > > > > Given that it is a huge effort, I am wondering if it would be sufficient > > to just to this change for 4.0 release though, and not translate any > > older versions at all? > > > > > > For including the docs in the release, I am actually wondering what the > > value is? Does anybody actually use these artifact? Of course, we need > > to publish JavaDocs, but the content of the web page seems to have very > > low value to the community? -- I did also check ASF guidelines about it > > and the webpage [1] say: > > > > > This material may include documentation concerning the release [...] > > > > So it is optional, but not required to publish docs artifacts from my > > understanding. I have no objections to keep publishing the web page > > docs, just saying, if the effort is too large and the value to low, we > > could consider just dropping it. > > > > > > I am also in favor of having easy access to `trunk / SNAPSHOT` version > > of the docs if possible. Some other ASF project do this already, for > > example Flink [2]. It's much easier to work with, and allows to spot > > issues during development. While it does not help on a PR directly, it > > is still a good way to verify a PR after it was merged, so any errors > > can be detected and corrected more easily. > > > > > > Also agree to the other raised points about generated content and > > permalinks. For permalinks, if effort is reasonable, I would rather try > > to redirect as many as possible. > > > > > > > > -Matthias > > > > > > > > > > [1] > > > https://www.apache.org/legal/release-policy.html#distribute-other-artifacts > > [2] https://nightlies.apache.org/flink/flink-docs-master/ > > > > > > On 2/25/25 8:43 AM, David Arthur wrote: > > > MM1: I suspect Hugo has some way of doing url redirection, but at the > > very > > > least we can add redirects in our htaccess file > > > https://github.com/apache/kafka-site/blob/asf-site/.htaccess > > > > > > Excluding the "front matter" (intro, quickstart, powered by, etc) I > think > > > the main links we need to worry about are the configuration docs pages. > > In > > > the demo, it looks like we are using the same anchors for specific > config > > > sections. > > > > > > E.g., > > > > > > https://kafka-site-md-711970345036.us-central1.run.app/39/configuration/configuration/#topicconfigs_compression.gzip.level > > > is the same content as > > > > > > https://kafka.apache.org/documentation/#topicconfigs_compression.zstd.level > > > > > > So maybe it shouldn't be too hard? > > > > > > Personally, I think the only links we should consider to be > "permalinks" > > > are the front matter and the documentation pages with the version in > the > > > URL. The unversioned documentation pages (like > > > > > > https://kafka.apache.org/documentation/#topicconfigs_compression.zstd.level > > ) > > > can change from release to release, so they are not expected to be > > > permanent. > > > > > > Do we have any data on inbound links to the site? Maybe from Google > > > Analytics or similar? > > > > > > -David A > > > > > > > > > > > > On Mon, Feb 24, 2025 at 10:17 AM Mickael Maison < > > mickael.mai...@gmail.com> > > > wrote: > > > > > >> Hi Harish, > > >> > > >> This is definitively something we want to do! > > >> > > >> MM1. One aspect to keep in mind in terms of compatibility is whether > > >> existing links will still work. Thousands of online resources link to > > >> the Apache Kafka website. Is it possible to keep all links working? > > >> > > >> MM2. A bunch of sections are generated. To do so we have a number of > > >> classes in the public API (for example 0, 1) with toHtml() or main() > > >> methods that output HTML. Should we update these to output Markdown > > >> directly instead? > > >> > > >> Converting the website has been discussed several times, I'd suggest > > >> checking at least https://issues.apache.org/jira/browse/KAFKA-2967 to > > >> see what was previously said. > > >> > > >> 0: > > >> > > > https://kafka.apache.org/39/javadoc/org/apache/kafka/common/config/ConfigDef.html#toHtml() > > >> 1: > > >> > > > https://kafka.apache.org/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > >> > > >> Thanks, > > >> Mickael > > >> > > >> On Mon, Feb 24, 2025 at 3:33 PM Bruno Cadonna <cado...@apache.org> > > wrote: > > >>> > > >>> Hi Harish, > > >>> > > >>> Thanks for the KIP! Great initiative! > > >>> > > >>> Do you plan to have a development version of the website online in > > >>> parallel to the old HTML version? > > >>> > > >>> I understand that locally with the markdown files and an markdown > > >>> editor, we see the rough structure of the docs (which is already an > > >>> improvement), but we will not see the end product, because for that > we > > >>> again need a web server. > > >>> > > >>> I assume that after you migrated all docs to markdown, we want to see > > >>> the complete website, review the content, and make some refinements. > > >>> > > >>> Best, > > >>> Bruno > > >>> > > >>> On 23.02.25 07:52, Harish Vishwanath wrote: > > >>>> Thank you David. Appreciate your feedback. My comments below: > > >>>> > > >>>> DA1) I think this is a very good suggestion. While building the > > >> prototype, > > >>>> I did spend additional effort for older versions of the doc since > > >> there are > > >>>> fairly significant differences in the structure and format of the > > >> documents > > >>>> as compared to 3.x versions. I have written the automation which can > > do > > >>>> this, but I do think hosting the older versions (<3.x) in its > current > > >>>> format and moving the rest to markdown will reduce the overall > > >>>> testing/validation effort. That said, if we foresee making > relatively > > >>>> frequent updates to older versions, it might be worth converting > > >> everything > > >>>> to markdown. > > >>>> > > >>>> DA2) My hunch is that this can be quite involved. Rendering a usable > > >> HTML > > >>>> version requires all the supporting artifacts (css, js, images, hugo > > >>>> shortcodes, partials etc.,) which will only be present in the > website > > >>>> documentation. Further, the hyper links themselves will not work > > >> without > > >>>> a webserver. To serve a directory structure based HTML, it will > > >> require us > > >>>> to have different templates for these docs where dependencies on the > > >>>> artifacts I mentioned above will need to be removed, links will need > > >> to be > > >>>> converted to use relative directory locations etc., > > >>>> > > >>>> DA3) This is definitely possible. Using tools such as `pandoc` we > > >> should be > > >>>> able to render plain text versions of markdown files. For ex, the > > below > > >>>> would create a plain text version of all docs for 39/ branch., which > > >> can be > > >>>> packaged with the binary distribution. > > >>>> > > >>>> find docs/ -name "*.md" -type f | while read -r file; do > > >>>> dir=$(dirname "$file") > > >>>> filename=$(basename "$file" .md) > > >>>> mkdir -p "39-ascii/$dir" > > >>>> pandoc -f markdown -t plain "$file" -o > > >> "39-ascii/$dir/$filename.txt" > > >>>> done > > >>>> > > >>>> DA4) Makes sense. I updated the wiki to reflect this. > > >>>> > > >>>> > > >>>> Regards, > > >>>> Harish > > >>>> > > >>>> > > >>>> On Fri, Feb 21, 2025 at 8:56 AM David Arthur <mum...@gmail.com> > > wrote: > > >>>> > > >>>>> Thanks for the KIP, Harish! > > >>>>> > > >>>>> I am +100 just on the title alone :) > > >>>>> > > >>>>> The motivations in the KIP are sound. All of these are very real > > >> annoyances > > >>>>> when it comes to maintaining our docs. I suspect we have not done > > >> anything > > >>>>> to improve them since the system we have worked well enough in the > > >> past. A > > >>>>> better way to say that is: it hasn't been bad enough to motivate us > > to > > >>>>> change it. > > >>>>> > > >>>>> With efforts like this, there is always some question about which > > >> tool or > > >>>>> method we should use. I like that your proposal includes these > > >> specifics as > > >>>>> well as a complete and functional demo. > > >>>>> > > >>>>> Few questions: > > >>>>> > > >>>>> DA1) Do we really need to convert all of our docs to the new thing? > > It > > >>>>> might be reasonable to draw a line somewhere (maybe 3.0?) and have > > >> docs > > >>>>> prior to that be hosted elsewhere (kafka.apache.org/old or > > >> something). > > >>>>> This > > >>>>> may or may not be more work than just converting everything. WDYT? > > >>>>> > > >>>>> DA2) For our packaged docs (those that go in the binaries we > > >> distribute), > > >>>>> do you think it's possible to render HTML that can be viewed > without > > a > > >>>>> webserver? Currently our packaged docs are kind of useless IMO. It > > >> would be > > >>>>> better if they could be viewed without any extra effort by an > > >> operator. > > >>>>> > > >>>>> DA3) As a possible alternative to DA2, could we render the markdown > > >> docs as > > >>>>> ascii text files? This is arguably better for operators since they > > >> don't > > >>>>> always have a GUI or web browser handy. > > >>>>> > > >>>>> DA4) Just a nitpick, could you reduce the "Sample directory layout" > > >> down to > > >>>>> one or two examples? It's difficult to see the versioned vs > > >> non-versioned > > >>>>> docs in the current block > > >>>>> > > >>>>> Thanks! > > >>>>> David A > > >>>>> > > >>>>> > > >>>>> On Sun, Feb 16, 2025 at 2:10 PM Harish Vishwanath < > > >>>>> harish.shas...@gmail.com> > > >>>>> wrote: > > >>>>> > > >>>>>> Hello, > > >>>>>> > > >>>>>> Following up from my previous email regarding moving AK > > >> documentation to > > >>>>>> markdown, I wrote KIP-1133 > > >>>>>> < > > >>>>>> > > >>>>> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown > > >>>>>>> > > >>>>>> detailing > > >>>>>> the proposal and would like to start discussions. Please take a > > look. > > >>>>>> > > >>>>>> Regards, > > >>>>>> Harish > > >>>>>> > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> David Arthur > > >>>>> > > >>>> > > >>> > > >> > > > > > > > > > > >
