Hi devs, Flink's Configuration page [1] has grown intimidatingly long and complex. Options are described across three main sections: common options (single section), advanced options (multiple sections), and full reference. The trailing "background" section further describes the most impactful options in much greater detail.
Several recent tickets, and a few outstanding, have added missing options to the configuration documentation. I'd like to propose a goal of organizing all options in the full reference into alphabetized, tabular form (one table per section), much like the system metrics [2]. Columns would be option name, description, and default value. The common and advanced sections could also be converted to tabular form with the exception of Kerberos-based Security. Missing options would be added to the full reference. Lastly, the simple heuristic for configuring network buffers has prompted many questions on the mailing list. With the 1.3 release the total and number of available buffers is reported through metrics and in the web dashboard. My experience has been that the number of required buffers is highly dependent on job topology and cluster performance. I propose keeping the simple heuristic and description while directing users to monitor the balance of available buffers. Greg [1] https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html [2] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#system-metrics [3] https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#configuring-the-network-buffers
