Hello, I'm planning to initiate a vote on this KIP next week. In the meantime, please share any further questions or comments you may have.
Regards, Harish On Tue, Mar 11, 2025 at 7:42 AM David Arthur <mum...@gmail.com> wrote: > TaiJuWu, > > Hugo has a Docker image which makes it very easy to run locally, and I > expect many developers have Docker installed already. I'm not totally sure, > but I don't think this would require any build support (Gradle or Hugo/Go). > I believe you would simply mount the docs directory in the docker container > and it would work. This is the workflow I expect to use since I don't plan > on managing a Go installation locally. > > -David > > On Tue, Mar 11, 2025 at 2:36 AM TaiJu Wu <tjwu1...@gmail.com> wrote: > > > Hi Harish, > > > > Thanks for this KIP. > > > > Is it possible to add dev website from trunk for developers? > > It could help developer to browser the web without local build and check > > recent documentation change quickly. > > I think it can improve our documentation a lot since developers can easy > to > > check it. > > > > Best, > > TaiJuWu > > > > > > Harish Vishwanath <harish.shas...@gmail.com> 於 2025年3月11日 週二 上午11:53寫道： > > > > > Thanks Chia-Ping. I have updated > > > < > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown#KIP1133:AKDocumentationandWebsiteinMarkdown-DevelopmentWorkflow > > > > > > > the KIP to include the above information. > > > > > > Regards, > > > Harish > > > > > > > > > On Sun, Mar 9, 2025 at 10:09 PM Chia-Ping Tsai <chia7...@gmail.com> > > wrote: > > > > > > > hi Harish > > > > > > > > > Hugo supports live-reload. The docsy theme has dependencies on > golang > > > and > > > > a > > > > couple of nodeJS libraries (autoprefixer, postcss etc.,) and as such > it > > > > will be simpler to use docker based workflows to avoid having to > > install > > > > any dependencies locally for development (ref: > > > > > > > > > > > > > > https://github.com/hvishwanath/kafka-site-md/blob/main/README.md#local-development-server > > > > ). We can set up CI pipelines to build and push production images > > serving > > > > the site with nginx. > > > > > > > > Thank you for the explanation. Would you please consider including > this > > > > information in the KIP? Documenting the new workflow for reviewing > > > > documentation would be very beneficial for future reference. In fact, > > we > > > > recently resolved a bug where the local site's behavior differed from > > the > > > > deployed version on Apache ... > > > > > > > > Best, > > > > Chia-Ping > > > > > > > > Harish Vishwanath <harish.shas...@gmail.com> 於 2025年3月10日 週一 > > 下午12:51寫道： > > > > > > > > > Thank you Chia-Ping. > > > > > > > > > > Conversion is mostly automated ( > > https://github.com/hvishwanath/ak2md > > > ) > > > > > and > > > > > as such the prototype > > > > > <https://kafka-site-md-711970345036.us-central1.run.app/> supports > > all > > > > > previous versions. However, not having to support all previous > > versions > > > > > will definitely be simpler. > > > > > > > > > > Hugo supports live-reload. The docsy theme has dependencies on > golang > > > > and a > > > > > couple of nodeJS libraries (autoprefixer, postcss etc.,) and as > such > > it > > > > > will be simpler to use docker based workflows to avoid having to > > > install > > > > > any dependencies locally for development (ref: > > > > > > > > > > > > > > > > > > > > https://github.com/hvishwanath/kafka-site-md/blob/main/README.md#local-development-server > > > > > ). We can set up CI pipelines to build and push production images > > > serving > > > > > the site with nginx. > > > > > > > > > > Regards, > > > > > Harish > > > > > > > > > > > > > > > On Sun, Mar 9, 2025 at 9:08 PM Chia-Ping Tsai <chia7...@gmail.com> > > > > wrote: > > > > > > > > > > > hi Harish > > > > > > > > > > > > I have to say, this KIP is one of the best ideas I've seen > > recently, > > > > and > > > > > I > > > > > > agree with Matthias that we don't need to translate any older > > > > versions. I > > > > > > truly hope this KIP can be included in 4.0. > > > > > > > > > > > > I have a quick question: what is the recommended procedure for > > > testing > > > > > the > > > > > > site locally after migrating to Markdown? Previously, we used a > > > > container > > > > > > to host the site locally for previewing changes. Could you please > > > > provide > > > > > > guidance on the updated workflow? > > > > > > > > > > > > Best, > > > > > > Chia-Ping > > > > > > > > > > > > Harish Vishwanath <harish.shas...@gmail.com> 於 2025年3月10日 週一 > > > > 上午11:53寫道： > > > > > > > > > > > > > Hello all > > > > > > > > > > > > > > Sorry for my delayed response. Thanks for your feedback and > > > comments. > > > > > > > Please see below: > > > > > > > > > > > > > > > Do you plan to have a development version of the website > online > > > in > > > > > > > parallel to the old HTML version? > > > > > > > Yes. The current prototype which supports all versions until AK > > 3.9 > > > > is > > > > > > > running at > > https://kafka-site-md-711970345036.us-central1.run.app/ > > > , > > > > > the > > > > > > > prototype > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown#KIP1133:AKDocumentationandWebsiteinMarkdown-Prototype > > > > > > > > > > > > > > > section of the KIP has more details. We should be able to host > > this > > > > in > > > > > > ASF > > > > > > > infra. Further, as described in the KIP > > > > > > > > > > > > > > >I understand that locally with the markdown files and an > > markdown > > > > > > editor, > > > > > > > we see the rough structure of the docs (which is already an > > > > > improvement), > > > > > > > but we will not see the end product, because for that we again > > > need a > > > > > web > > > > > > > server. > > > > > > > Hugo supports live reload making local development easier (ref: > > > > > > > > > > https://github.com/hvishwanath/kafka-site-md/blob/main/Makefile#L25) > > > > > > > > > > > > > > > > > > > > > [MM1] Yes., I have ensured that all existing auto generated > > > > > > documentations > > > > > > > (html files under javadoc, generated folders) are available at > > the > > > > same > > > > > > > relative location as they are in the current website. You can > see > > > > this > > > > > in > > > > > > > the prototype. > > > > > > > For example: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > > > > > is > > > > > > > available at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kafka-site-md-711970345036.us-central1.run.app/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kafka-site-md-711970345036.us-central1.run.app/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > > > > > > > > > > > > > Furthermore, for production, we can serve this with a full > blown > > > web > > > > > > server > > > > > > > such as nginx allowing us to set up any required redirects. > > > > > > > > > > > > > > [MM2] I recommend that we leave any auto generated > documentation > > to > > > > > > > continue to be in HTML format. We can serve them as static > assets > > > > (ref: > > > > > > > https://github.com/hvishwanath/kafka-site-md/tree/main/static) > > > > > > > > > > > > > > >I am also in favor of having easy access to `trunk / SNAPSHOT` > > > > version > > > > > > of > > > > > > > the docs if possible. > > > > > > > Yes, this is definitely possible. > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > Harish > > > > > > > > > > > > > > > > > > > > > On Sat, Mar 1, 2025 at 3:01 PM Matthias J. Sax < > mj...@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > I am totally in favor to make this happen, and I was > > disappointed > > > > > each > > > > > > > > time in the past, when it was proposed but we did not do it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Given that it is a huge effort, I am wondering if it would be > > > > > > sufficient > > > > > > > > to just to this change for 4.0 release though, and not > > translate > > > > any > > > > > > > > older versions at all? > > > > > > > > > > > > > > > > > > > > > > > > For including the docs in the release, I am actually > wondering > > > what > > > > > the > > > > > > > > value is? Does anybody actually use these artifact? Of > course, > > we > > > > > need > > > > > > > > to publish JavaDocs, but the content of the web page seems to > > > have > > > > > very > > > > > > > > low value to the community? -- I did also check ASF > guidelines > > > > about > > > > > it > > > > > > > > and the webpage [1] say: > > > > > > > > > > > > > > > > > This material may include documentation concerning the > > release > > > > > [...] > > > > > > > > > > > > > > > > So it is optional, but not required to publish docs artifacts > > > from > > > > my > > > > > > > > understanding. I have no objections to keep publishing the > web > > > page > > > > > > > > docs, just saying, if the effort is too large and the value > to > > > low, > > > > > we > > > > > > > > could consider just dropping it. > > > > > > > > > > > > > > > > > > > > > > > > I am also in favor of having easy access to `trunk / > SNAPSHOT` > > > > > version > > > > > > > > of the docs if possible. Some other ASF project do this > > already, > > > > for > > > > > > > > example Flink [2]. It's much easier to work with, and allows > to > > > > spot > > > > > > > > issues during development. While it does not help on a PR > > > directly, > > > > > it > > > > > > > > is still a good way to verify a PR after it was merged, so > any > > > > errors > > > > > > > > can be detected and corrected more easily. > > > > > > > > > > > > > > > > > > > > > > > > Also agree to the other raised points about generated content > > and > > > > > > > > permalinks. For permalinks, if effort is reasonable, I would > > > rather > > > > > try > > > > > > > > to redirect as many as possible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Matthias > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.apache.org/legal/release-policy.html#distribute-other-artifacts > > > > > > > > [2] https://nightlies.apache.org/flink/flink-docs-master/ > > > > > > > > > > > > > > > > > > > > > > > > On 2/25/25 8:43 AM, David Arthur wrote: > > > > > > > > > MM1: I suspect Hugo has some way of doing url redirection, > > but > > > at > > > > > the > > > > > > > > very > > > > > > > > > least we can add redirects in our htaccess file > > > > > > > > > > https://github.com/apache/kafka-site/blob/asf-site/.htaccess > > > > > > > > > > > > > > > > > > Excluding the "front matter" (intro, quickstart, powered > by, > > > > etc) I > > > > > > > think > > > > > > > > > the main links we need to worry about are the configuration > > > docs > > > > > > pages. > > > > > > > > In > > > > > > > > > the demo, it looks like we are using the same anchors for > > > > specific > > > > > > > config > > > > > > > > > sections. > > > > > > > > > > > > > > > > > > E.g., > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kafka-site-md-711970345036.us-central1.run.app/39/configuration/configuration/#topicconfigs_compression.gzip.level > > > > > > > > > is the same content as > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/documentation/#topicconfigs_compression.zstd.level > > > > > > > > > > > > > > > > > > So maybe it shouldn't be too hard? > > > > > > > > > > > > > > > > > > Personally, I think the only links we should consider to be > > > > > > > "permalinks" > > > > > > > > > are the front matter and the documentation pages with the > > > version > > > > > in > > > > > > > the > > > > > > > > > URL. The unversioned documentation pages (like > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/documentation/#topicconfigs_compression.zstd.level > > > > > > > > ) > > > > > > > > > can change from release to release, so they are not > expected > > to > > > > be > > > > > > > > > permanent. > > > > > > > > > > > > > > > > > > Do we have any data on inbound links to the site? Maybe > from > > > > Google > > > > > > > > > Analytics or similar? > > > > > > > > > > > > > > > > > > -David A > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Feb 24, 2025 at 10:17 AM Mickael Maison < > > > > > > > > mickael.mai...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> Hi Harish, > > > > > > > > >> > > > > > > > > >> This is definitively something we want to do! > > > > > > > > >> > > > > > > > > >> MM1. One aspect to keep in mind in terms of compatibility > is > > > > > whether > > > > > > > > >> existing links will still work. Thousands of online > > resources > > > > link > > > > > > to > > > > > > > > >> the Apache Kafka website. Is it possible to keep all links > > > > > working? > > > > > > > > >> > > > > > > > > >> MM2. A bunch of sections are generated. To do so we have a > > > > number > > > > > of > > > > > > > > >> classes in the public API (for example 0, 1) with toHtml() > > or > > > > > main() > > > > > > > > >> methods that output HTML. Should we update these to output > > > > > Markdown > > > > > > > > >> directly instead? > > > > > > > > >> > > > > > > > > >> Converting the website has been discussed several times, > I'd > > > > > suggest > > > > > > > > >> checking at least > > > > > https://issues.apache.org/jira/browse/KAFKA-2967 > > > > > > to > > > > > > > > >> see what was previously said. > > > > > > > > >> > > > > > > > > >> 0: > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/39/javadoc/org/apache/kafka/common/config/ConfigDef.html#toHtml() > > > > > > > > >> 1: > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/39/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html#main(java.lang.String%5B%5D) > > > > > > > > >> > > > > > > > > >> Thanks, > > > > > > > > >> Mickael > > > > > > > > >> > > > > > > > > >> On Mon, Feb 24, 2025 at 3:33 PM Bruno Cadonna < > > > > cado...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > >>> > > > > > > > > >>> Hi Harish, > > > > > > > > >>> > > > > > > > > >>> Thanks for the KIP! Great initiative! > > > > > > > > >>> > > > > > > > > >>> Do you plan to have a development version of the website > > > online > > > > > in > > > > > > > > >>> parallel to the old HTML version? > > > > > > > > >>> > > > > > > > > >>> I understand that locally with the markdown files and an > > > > markdown > > > > > > > > >>> editor, we see the rough structure of the docs (which is > > > > already > > > > > an > > > > > > > > >>> improvement), but we will not see the end product, > because > > > for > > > > > that > > > > > > > we > > > > > > > > >>> again need a web server. > > > > > > > > >>> > > > > > > > > >>> I assume that after you migrated all docs to markdown, we > > > want > > > > to > > > > > > see > > > > > > > > >>> the complete website, review the content, and make some > > > > > > refinements. > > > > > > > > >>> > > > > > > > > >>> Best, > > > > > > > > >>> Bruno > > > > > > > > >>> > > > > > > > > >>> On 23.02.25 07:52, Harish Vishwanath wrote: > > > > > > > > >>>> Thank you David. Appreciate your feedback. My comments > > > below: > > > > > > > > >>>> > > > > > > > > >>>> DA1) I think this is a very good suggestion. While > > building > > > > the > > > > > > > > >> prototype, > > > > > > > > >>>> I did spend additional effort for older versions of the > > doc > > > > > since > > > > > > > > >> there are > > > > > > > > >>>> fairly significant differences in the structure and > format > > > of > > > > > the > > > > > > > > >> documents > > > > > > > > >>>> as compared to 3.x versions. I have written the > automation > > > > which > > > > > > can > > > > > > > > do > > > > > > > > >>>> this, but I do think hosting the older versions (<3.x) > in > > > its > > > > > > > current > > > > > > > > >>>> format and moving the rest to markdown will reduce the > > > overall > > > > > > > > >>>> testing/validation effort. That said, if we foresee > making > > > > > > > relatively > > > > > > > > >>>> frequent updates to older versions, it might be worth > > > > converting > > > > > > > > >> everything > > > > > > > > >>>> to markdown. > > > > > > > > >>>> > > > > > > > > >>>> DA2) My hunch is that this can be quite involved. > > Rendering > > > a > > > > > > usable > > > > > > > > >> HTML > > > > > > > > >>>> version requires all the supporting artifacts (css, js, > > > > images, > > > > > > hugo > > > > > > > > >>>> shortcodes, partials etc.,) which will only be present > in > > > the > > > > > > > website > > > > > > > > >>>> documentation. Further, the hyper links themselves will > > not > > > > work > > > > > > > > >> without > > > > > > > > >>>> a webserver. To serve a directory structure based HTML, > it > > > > will > > > > > > > > >> require us > > > > > > > > >>>> to have different templates for these docs where > > > dependencies > > > > on > > > > > > the > > > > > > > > >>>> artifacts I mentioned above will need to be removed, > links > > > > will > > > > > > need > > > > > > > > >> to be > > > > > > > > >>>> converted to use relative directory locations etc., > > > > > > > > >>>> > > > > > > > > >>>> DA3) This is definitely possible. Using tools such as > > > `pandoc` > > > > > we > > > > > > > > >> should be > > > > > > > > >>>> able to render plain text versions of markdown files. > For > > > ex, > > > > > the > > > > > > > > below > > > > > > > > >>>> would create a plain text version of all docs for 39/ > > > branch., > > > > > > which > > > > > > > > >> can be > > > > > > > > >>>> packaged with the binary distribution. > > > > > > > > >>>> > > > > > > > > >>>> find docs/ -name "*.md" -type f | while read -r file; do > > > > > > > > >>>> dir=$(dirname "$file") > > > > > > > > >>>> filename=$(basename "$file" .md) > > > > > > > > >>>> mkdir -p "39-ascii/$dir" > > > > > > > > >>>> pandoc -f markdown -t plain "$file" -o > > > > > > > > >> "39-ascii/$dir/$filename.txt" > > > > > > > > >>>> done > > > > > > > > >>>> > > > > > > > > >>>> DA4) Makes sense. I updated the wiki to reflect this. > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> Regards, > > > > > > > > >>>> Harish > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> On Fri, Feb 21, 2025 at 8:56 AM David Arthur < > > > > mum...@gmail.com> > > > > > > > > wrote: > > > > > > > > >>>> > > > > > > > > >>>>> Thanks for the KIP, Harish! > > > > > > > > >>>>> > > > > > > > > >>>>> I am +100 just on the title alone :) > > > > > > > > >>>>> > > > > > > > > >>>>> The motivations in the KIP are sound. All of these are > > very > > > > > real > > > > > > > > >> annoyances > > > > > > > > >>>>> when it comes to maintaining our docs. I suspect we > have > > > not > > > > > done > > > > > > > > >> anything > > > > > > > > >>>>> to improve them since the system we have worked well > > enough > > > > in > > > > > > the > > > > > > > > >> past. A > > > > > > > > >>>>> better way to say that is: it hasn't been bad enough to > > > > > motivate > > > > > > us > > > > > > > > to > > > > > > > > >>>>> change it. > > > > > > > > >>>>> > > > > > > > > >>>>> With efforts like this, there is always some question > > about > > > > > which > > > > > > > > >> tool or > > > > > > > > >>>>> method we should use. I like that your proposal > includes > > > > these > > > > > > > > >> specifics as > > > > > > > > >>>>> well as a complete and functional demo. > > > > > > > > >>>>> > > > > > > > > >>>>> Few questions: > > > > > > > > >>>>> > > > > > > > > >>>>> DA1) Do we really need to convert all of our docs to > the > > > new > > > > > > thing? > > > > > > > > It > > > > > > > > >>>>> might be reasonable to draw a line somewhere (maybe > 3.0?) > > > and > > > > > > have > > > > > > > > >> docs > > > > > > > > >>>>> prior to that be hosted elsewhere ( > kafka.apache.org/old > > or > > > > > > > > >> something). > > > > > > > > >>>>> This > > > > > > > > >>>>> may or may not be more work than just converting > > > everything. > > > > > > WDYT? > > > > > > > > >>>>> > > > > > > > > >>>>> DA2) For our packaged docs (those that go in the > binaries > > > we > > > > > > > > >> distribute), > > > > > > > > >>>>> do you think it's possible to render HTML that can be > > > viewed > > > > > > > without > > > > > > > > a > > > > > > > > >>>>> webserver? Currently our packaged docs are kind of > > useless > > > > IMO. > > > > > > It > > > > > > > > >> would be > > > > > > > > >>>>> better if they could be viewed without any extra effort > > by > > > an > > > > > > > > >> operator. > > > > > > > > >>>>> > > > > > > > > >>>>> DA3) As a possible alternative to DA2, could we render > > the > > > > > > markdown > > > > > > > > >> docs as > > > > > > > > >>>>> ascii text files? This is arguably better for operators > > > since > > > > > > they > > > > > > > > >> don't > > > > > > > > >>>>> always have a GUI or web browser handy. > > > > > > > > >>>>> > > > > > > > > >>>>> DA4) Just a nitpick, could you reduce the "Sample > > directory > > > > > > layout" > > > > > > > > >> down to > > > > > > > > >>>>> one or two examples? It's difficult to see the > versioned > > vs > > > > > > > > >> non-versioned > > > > > > > > >>>>> docs in the current block > > > > > > > > >>>>> > > > > > > > > >>>>> Thanks! > > > > > > > > >>>>> David A > > > > > > > > >>>>> > > > > > > > > >>>>> > > > > > > > > >>>>> On Sun, Feb 16, 2025 at 2:10 PM Harish Vishwanath < > > > > > > > > >>>>> harish.shas...@gmail.com> > > > > > > > > >>>>> wrote: > > > > > > > > >>>>> > > > > > > > > >>>>>> Hello, > > > > > > > > >>>>>> > > > > > > > > >>>>>> Following up from my previous email regarding moving > AK > > > > > > > > >> documentation to > > > > > > > > >>>>>> markdown, I wrote KIP-1133 > > > > > > > > >>>>>> < > > > > > > > > >>>>>> > > > > > > > > >>>>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1133%3A+AK+Documentation+and+Website+in+Markdown > > > > > > > > >>>>>>> > > > > > > > > >>>>>> detailing > > > > > > > > >>>>>> the proposal and would like to start discussions. > Please > > > > take > > > > > a > > > > > > > > look. > > > > > > > > >>>>>> > > > > > > > > >>>>>> Regards, > > > > > > > > >>>>>> Harish > > > > > > > > >>>>>> > > > > > > > > >>>>> > > > > > > > > >>>>> > > > > > > > > >>>>> -- > > > > > > > > >>>>> David Arthur > > > > > > > > >>>>> > > > > > > > > >>>> > > > > > > > > >>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > David Arthur >
