Thanks Chia-Ping. I have updated <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown#KIP1133:AKDocumentationandWebsiteinMarkdown-DevelopmentWorkflow> the KIP to include the above information.
Regards, Harish On Sun, Mar 9, 2025 at 10:09 PM Chia-Ping Tsai <chia7...@gmail.com> wrote: > hi Harish > > > Hugo supports live-reload. The docsy theme has dependencies on golang and > a > couple of nodeJS libraries (autoprefixer, postcss etc.,) and as such it > will be simpler to use docker based workflows to avoid having to install > any dependencies locally for development (ref: > > https://github.com/hvishwanath/kafka-site-md/blob/main/README.md#local-development-server > ). We can set up CI pipelines to build and push production images serving > the site with nginx. > > Thank you for the explanation. Would you please consider including this > information in the KIP? Documenting the new workflow for reviewing > documentation would be very beneficial for future reference. In fact, we > recently resolved a bug where the local site's behavior differed from the > deployed version on Apache ... > > Best, > Chia-Ping > > Harish Vishwanath <harish.shas...@gmail.com> 於 2025年3月10日 週一 下午12:51寫道： > > > Thank you Chia-Ping. > > > > Conversion is mostly automated ( https://github.com/hvishwanath/ak2md ) > > and > > as such the prototype > > <https://kafka-site-md-711970345036.us-central1.run.app/> supports all > > previous versions. However, not having to support all previous versions > > will definitely be simpler. > > > > Hugo supports live-reload. The docsy theme has dependencies on golang > and a > > couple of nodeJS libraries (autoprefixer, postcss etc.,) and as such it > > will be simpler to use docker based workflows to avoid having to install > > any dependencies locally for development (ref: > > > > > https://github.com/hvishwanath/kafka-site-md/blob/main/README.md#local-development-server > > ). We can set up CI pipelines to build and push production images serving > > the site with nginx. > > > > Regards, > > Harish > > > > > > On Sun, Mar 9, 2025 at 9:08 PM Chia-Ping Tsai <chia7...@gmail.com> > wrote: > > > > > hi Harish > > > > > > I have to say, this KIP is one of the best ideas I've seen recently, > and > > I > > > agree with Matthias that we don't need to translate any older > versions. I > > > truly hope this KIP can be included in 4.0. > > > > > > I have a quick question: what is the recommended procedure for testing > > the > > > site locally after migrating to Markdown? Previously, we used a > container > > > to host the site locally for previewing changes. Could you please > provide > > > guidance on the updated workflow? > > > > > > Best, > > > Chia-Ping > > > > > > Harish Vishwanath <harish.shas...@gmail.com> 於 2025年3月10日 週一 > 上午11:53寫道： > > > > > > > Hello all > > > > > > > > Sorry for my delayed response. Thanks for your feedback and comments. > > > > Please see below: > > > > > > > > > Do you plan to have a development version of the website online in > > > > parallel to the old HTML version? > > > > Yes. The current prototype which supports all versions until AK 3.9 > is > > > > running at https://kafka-site-md-711970345036.us-central1.run.app/, > > the > > > > prototype > > > > < > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown#KIP1133:AKDocumentationandWebsiteinMarkdown-Prototype > > > > > > > > > section of the KIP has more details. We should be able to host this > in > > > ASF > > > > infra. Further, as described in the KIP > > > > > > > > >I understand that locally with the markdown files and an markdown > > > editor, > > > > we see the rough structure of the docs (which is already an > > improvement), > > > > but we will not see the end product, because for that we again need a > > web > > > > server. > > > > Hugo supports live reload making local development easier (ref: > > > > https://github.com/hvishwanath/kafka-site-md/blob/main/Makefile#L25) > > > > > > > > > > > > [MM1] Yes., I have ensured that all existing auto generated > > > documentations > > > > (html files under javadoc, generated folders) are available at the > same > > > > relative location as they are in the current website. You can see > this > > in > > > > the prototype. > > > > For example: > > > > > > > > > > > > > > https://kafka.apache.org/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > > is > > > > available at > > > > > > > > > > > > > > https://kafka-site-md-711970345036.us-central1.run.app/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > > < > > > > > > > > > > https://kafka-site-md-711970345036.us-central1.run.app/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > > > > > > > Furthermore, for production, we can serve this with a full blown web > > > server > > > > such as nginx allowing us to set up any required redirects. > > > > > > > > [MM2] I recommend that we leave any auto generated documentation to > > > > continue to be in HTML format. We can serve them as static assets > (ref: > > > > https://github.com/hvishwanath/kafka-site-md/tree/main/static) > > > > > > > > >I am also in favor of having easy access to `trunk / SNAPSHOT` > version > > > of > > > > the docs if possible. > > > > Yes, this is definitely possible. > > > > > > > > > > > > Regards, > > > > Harish > > > > > > > > > > > > On Sat, Mar 1, 2025 at 3:01 PM Matthias J. Sax <mj...@apache.org> > > wrote: > > > > > > > > > Hi, > > > > > > > > > > I am totally in favor to make this happen, and I was disappointed > > each > > > > > time in the past, when it was proposed but we did not do it. > > > > > > > > > > > > > > > > > > > > Given that it is a huge effort, I am wondering if it would be > > > sufficient > > > > > to just to this change for 4.0 release though, and not translate > any > > > > > older versions at all? > > > > > > > > > > > > > > > For including the docs in the release, I am actually wondering what > > the > > > > > value is? Does anybody actually use these artifact? Of course, we > > need > > > > > to publish JavaDocs, but the content of the web page seems to have > > very > > > > > low value to the community? -- I did also check ASF guidelines > about > > it > > > > > and the webpage [1] say: > > > > > > > > > > > This material may include documentation concerning the release > > [...] > > > > > > > > > > So it is optional, but not required to publish docs artifacts from > my > > > > > understanding. I have no objections to keep publishing the web page > > > > > docs, just saying, if the effort is too large and the value to low, > > we > > > > > could consider just dropping it. > > > > > > > > > > > > > > > I am also in favor of having easy access to `trunk / SNAPSHOT` > > version > > > > > of the docs if possible. Some other ASF project do this already, > for > > > > > example Flink [2]. It's much easier to work with, and allows to > spot > > > > > issues during development. While it does not help on a PR directly, > > it > > > > > is still a good way to verify a PR after it was merged, so any > errors > > > > > can be detected and corrected more easily. > > > > > > > > > > > > > > > Also agree to the other raised points about generated content and > > > > > permalinks. For permalinks, if effort is reasonable, I would rather > > try > > > > > to redirect as many as possible. > > > > > > > > > > > > > > > > > > > > -Matthias > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://www.apache.org/legal/release-policy.html#distribute-other-artifacts > > > > > [2] https://nightlies.apache.org/flink/flink-docs-master/ > > > > > > > > > > > > > > > On 2/25/25 8:43 AM, David Arthur wrote: > > > > > > MM1: I suspect Hugo has some way of doing url redirection, but at > > the > > > > > very > > > > > > least we can add redirects in our htaccess file > > > > > > https://github.com/apache/kafka-site/blob/asf-site/.htaccess > > > > > > > > > > > > Excluding the "front matter" (intro, quickstart, powered by, > etc) I > > > > think > > > > > > the main links we need to worry about are the configuration docs > > > pages. > > > > > In > > > > > > the demo, it looks like we are using the same anchors for > specific > > > > config > > > > > > sections. > > > > > > > > > > > > E.g., > > > > > > > > > > > > > > > > > > > > > https://kafka-site-md-711970345036.us-central1.run.app/39/configuration/configuration/#topicconfigs_compression.gzip.level > > > > > > is the same content as > > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/documentation/#topicconfigs_compression.zstd.level > > > > > > > > > > > > So maybe it shouldn't be too hard? > > > > > > > > > > > > Personally, I think the only links we should consider to be > > > > "permalinks" > > > > > > are the front matter and the documentation pages with the version > > in > > > > the > > > > > > URL. The unversioned documentation pages (like > > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/documentation/#topicconfigs_compression.zstd.level > > > > > ) > > > > > > can change from release to release, so they are not expected to > be > > > > > > permanent. > > > > > > > > > > > > Do we have any data on inbound links to the site? Maybe from > Google > > > > > > Analytics or similar? > > > > > > > > > > > > -David A > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 24, 2025 at 10:17 AM Mickael Maison < > > > > > mickael.mai...@gmail.com> > > > > > > wrote: > > > > > > > > > > > >> Hi Harish, > > > > > >> > > > > > >> This is definitively something we want to do! > > > > > >> > > > > > >> MM1. One aspect to keep in mind in terms of compatibility is > > whether > > > > > >> existing links will still work. Thousands of online resources > link > > > to > > > > > >> the Apache Kafka website. Is it possible to keep all links > > working? > > > > > >> > > > > > >> MM2. A bunch of sections are generated. To do so we have a > number > > of > > > > > >> classes in the public API (for example 0, 1) with toHtml() or > > main() > > > > > >> methods that output HTML. Should we update these to output > > Markdown > > > > > >> directly instead? > > > > > >> > > > > > >> Converting the website has been discussed several times, I'd > > suggest > > > > > >> checking at least > > https://issues.apache.org/jira/browse/KAFKA-2967 > > > to > > > > > >> see what was previously said. > > > > > >> > > > > > >> 0: > > > > > >> > > > > > > > > > > > > > > > https://kafka.apache.org/39/javadoc/org/apache/kafka/common/config/ConfigDef.html#toHtml() > > > > > >> 1: > > > > > >> > > > > > > > > > > > > > > > https://kafka.apache.org/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > > > >> > > > > > >> Thanks, > > > > > >> Mickael > > > > > >> > > > > > >> On Mon, Feb 24, 2025 at 3:33 PM Bruno Cadonna < > cado...@apache.org > > > > > > > > wrote: > > > > > >>> > > > > > >>> Hi Harish, > > > > > >>> > > > > > >>> Thanks for the KIP! Great initiative! > > > > > >>> > > > > > >>> Do you plan to have a development version of the website online > > in > > > > > >>> parallel to the old HTML version? > > > > > >>> > > > > > >>> I understand that locally with the markdown files and an > markdown > > > > > >>> editor, we see the rough structure of the docs (which is > already > > an > > > > > >>> improvement), but we will not see the end product, because for > > that > > > > we > > > > > >>> again need a web server. > > > > > >>> > > > > > >>> I assume that after you migrated all docs to markdown, we want > to > > > see > > > > > >>> the complete website, review the content, and make some > > > refinements. > > > > > >>> > > > > > >>> Best, > > > > > >>> Bruno > > > > > >>> > > > > > >>> On 23.02.25 07:52, Harish Vishwanath wrote: > > > > > >>>> Thank you David. Appreciate your feedback. My comments below: > > > > > >>>> > > > > > >>>> DA1) I think this is a very good suggestion. While building > the > > > > > >> prototype, > > > > > >>>> I did spend additional effort for older versions of the doc > > since > > > > > >> there are > > > > > >>>> fairly significant differences in the structure and format of > > the > > > > > >> documents > > > > > >>>> as compared to 3.x versions. I have written the automation > which > > > can > > > > > do > > > > > >>>> this, but I do think hosting the older versions (<3.x) in its > > > > current > > > > > >>>> format and moving the rest to markdown will reduce the overall > > > > > >>>> testing/validation effort. That said, if we foresee making > > > > relatively > > > > > >>>> frequent updates to older versions, it might be worth > converting > > > > > >> everything > > > > > >>>> to markdown. > > > > > >>>> > > > > > >>>> DA2) My hunch is that this can be quite involved. Rendering a > > > usable > > > > > >> HTML > > > > > >>>> version requires all the supporting artifacts (css, js, > images, > > > hugo > > > > > >>>> shortcodes, partials etc.,) which will only be present in the > > > > website > > > > > >>>> documentation. Further, the hyper links themselves will not > work > > > > > >> without > > > > > >>>> a webserver. To serve a directory structure based HTML, it > will > > > > > >> require us > > > > > >>>> to have different templates for these docs where dependencies > on > > > the > > > > > >>>> artifacts I mentioned above will need to be removed, links > will > > > need > > > > > >> to be > > > > > >>>> converted to use relative directory locations etc., > > > > > >>>> > > > > > >>>> DA3) This is definitely possible. Using tools such as `pandoc` > > we > > > > > >> should be > > > > > >>>> able to render plain text versions of markdown files. For ex, > > the > > > > > below > > > > > >>>> would create a plain text version of all docs for 39/ branch., > > > which > > > > > >> can be > > > > > >>>> packaged with the binary distribution. > > > > > >>>> > > > > > >>>> find docs/ -name "*.md" -type f | while read -r file; do > > > > > >>>> dir=$(dirname "$file") > > > > > >>>> filename=$(basename "$file" .md) > > > > > >>>> mkdir -p "39-ascii/$dir" > > > > > >>>> pandoc -f markdown -t plain "$file" -o > > > > > >> "39-ascii/$dir/$filename.txt" > > > > > >>>> done > > > > > >>>> > > > > > >>>> DA4) Makes sense. I updated the wiki to reflect this. > > > > > >>>> > > > > > >>>> > > > > > >>>> Regards, > > > > > >>>> Harish > > > > > >>>> > > > > > >>>> > > > > > >>>> On Fri, Feb 21, 2025 at 8:56 AM David Arthur < > mum...@gmail.com> > > > > > wrote: > > > > > >>>> > > > > > >>>>> Thanks for the KIP, Harish! > > > > > >>>>> > > > > > >>>>> I am +100 just on the title alone :) > > > > > >>>>> > > > > > >>>>> The motivations in the KIP are sound. All of these are very > > real > > > > > >> annoyances > > > > > >>>>> when it comes to maintaining our docs. I suspect we have not > > done > > > > > >> anything > > > > > >>>>> to improve them since the system we have worked well enough > in > > > the > > > > > >> past. A > > > > > >>>>> better way to say that is: it hasn't been bad enough to > > motivate > > > us > > > > > to > > > > > >>>>> change it. > > > > > >>>>> > > > > > >>>>> With efforts like this, there is always some question about > > which > > > > > >> tool or > > > > > >>>>> method we should use. I like that your proposal includes > these > > > > > >> specifics as > > > > > >>>>> well as a complete and functional demo. > > > > > >>>>> > > > > > >>>>> Few questions: > > > > > >>>>> > > > > > >>>>> DA1) Do we really need to convert all of our docs to the new > > > thing? > > > > > It > > > > > >>>>> might be reasonable to draw a line somewhere (maybe 3.0?) and > > > have > > > > > >> docs > > > > > >>>>> prior to that be hosted elsewhere (kafka.apache.org/old or > > > > > >> something). > > > > > >>>>> This > > > > > >>>>> may or may not be more work than just converting everything. > > > WDYT? > > > > > >>>>> > > > > > >>>>> DA2) For our packaged docs (those that go in the binaries we > > > > > >> distribute), > > > > > >>>>> do you think it's possible to render HTML that can be viewed > > > > without > > > > > a > > > > > >>>>> webserver? Currently our packaged docs are kind of useless > IMO. > > > It > > > > > >> would be > > > > > >>>>> better if they could be viewed without any extra effort by an > > > > > >> operator. > > > > > >>>>> > > > > > >>>>> DA3) As a possible alternative to DA2, could we render the > > > markdown > > > > > >> docs as > > > > > >>>>> ascii text files? This is arguably better for operators since > > > they > > > > > >> don't > > > > > >>>>> always have a GUI or web browser handy. > > > > > >>>>> > > > > > >>>>> DA4) Just a nitpick, could you reduce the "Sample directory > > > layout" > > > > > >> down to > > > > > >>>>> one or two examples? It's difficult to see the versioned vs > > > > > >> non-versioned > > > > > >>>>> docs in the current block > > > > > >>>>> > > > > > >>>>> Thanks! > > > > > >>>>> David A > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> On Sun, Feb 16, 2025 at 2:10 PM Harish Vishwanath < > > > > > >>>>> harish.shas...@gmail.com> > > > > > >>>>> wrote: > > > > > >>>>> > > > > > >>>>>> Hello, > > > > > >>>>>> > > > > > >>>>>> Following up from my previous email regarding moving AK > > > > > >> documentation to > > > > > >>>>>> markdown, I wrote KIP-1133 > > > > > >>>>>> < > > > > > >>>>>> > > > > > >>>>> > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown > > > > > >>>>>>> > > > > > >>>>>> detailing > > > > > >>>>>> the proposal and would like to start discussions. Please > take > > a > > > > > look. > > > > > >>>>>> > > > > > >>>>>> Regards, > > > > > >>>>>> Harish > > > > > >>>>>> > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> -- > > > > > >>>>> David Arthur > > > > > >>>>> > > > > > >>>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
