reiabreu commented on PR #8819: URL: https://github.com/apache/storm/pull/8819#issuecomment-4857349768
Hi Richard! I will have a closer look ASAP. But a quick note just looking at the description: a release will not have the storm-kafka-monitor out of the box; it will be pulled as needed at runtime. This is conditional on how organizations provision their systems and the type of connectivity those boxes have to the internet. A full-fledged binary distribution is the only way we can cover all cases, even if that means being overprovisioned. On Tue, 30 Jun 2026 at 19:16, Richard Zowalla ***@***.***> wrote: > What & why > > The binary distribution ships ~395 MB of jars, much of it optional or > duplicated. This PR trims it substantially without removing any capability: > optional pieces become fetch-on-demand, and jars shared between the daemon > and worker classpaths are de-duplicated. > > Smaller artifact means less download/storage/registry bandwidth and a > smaller container image, and a lighter CI/CD carbon footprint. 🌱 > Changes > > 1. > > *storm-autocreds no longer bundled (-79 MB).* It pulls the full > Hadoop/HBase client tree but is only used on secure (Kerberos) clusters and > is off by default. Now ships only the README (like the other external/* > connectors); bin/storm-autocreds-fetch retrieves the plugin and its > deps into extlib-daemon. > 2. > > *storm-kafka-monitor no longer bundled (-38 MB).* Only needed to show > Kafka spout lag in the UI or to run bin/storm-kafka-monitor. > bin/storm-kafka-monitor-fetch installs it into > lib-tools/storm-kafka-monitor. The UI degrades gracefully when it is > absent (TopologySpoutLag detects it, shows an actionable message, logs > once) and the wrapper prints a hint instead of ClassNotFound. > 3. > > *lib-common de-duplication (-71 MB).* The worker classpath (lib-worker) > is a byte-identical subset of the daemon classpath (lib). Shared jars > are now kept once in lib-common/ (added to both classpaths via > bin/storm.py) and removed from lib. dedup-libs.py only merges > byte-identical jars (name and sha-256), so no version is silently merged; > tool classpaths are left untouched. > > Net: roughly -188 MB (~47%) of the bundled jar payload, with no loss of > functionality. > Notes for reviewers > > Opening as draft to get early feedback on the approach. Still need to > verify on a Linux system (full -Pdist native distribution build). > ------------------------------ > You can view, comment on, or merge this pull request online at: > > https://github.com/apache/storm/pull/8819 > Commit Summary > > - 8cf5dd8 > <https://github.com/apache/storm/pull/8819/commits/8cf5dd818ac46b1c2d857967e9d89a9ae502c550> > build: stop bundling storm-autocreds in the binary distribution > - 696dbc7 > <https://github.com/apache/storm/pull/8819/commits/696dbc7e0020c6a8c9e20a1c6ec97012ce20d200> > build: stop bundling storm-kafka-monitor in the binary distribution > - 3aeac63 > <https://github.com/apache/storm/pull/8819/commits/3aeac63c8477307d1598917dd1d332c3534a9b69> > build: add lib-common to the daemon and worker classpaths > - 4bd2fda > <https://github.com/apache/storm/pull/8819/commits/4bd2fda52f088555a9e4c17a47527e1db329bfcd> > build: add dedup-libs script to share daemon/worker jars via lib-common > - 97d2922 > <https://github.com/apache/storm/pull/8819/commits/97d292229d0c994eb65747dcc12444a1c98dc85a> > build: wire lib-common de-duplication into the binary assembly > > File Changes > > (16 files <https://github.com/apache/storm/pull/8819/files>) > > - *A* bin/storm-autocreds-fetch > <https://github.com/apache/storm/pull/8819/files#diff-3e33fb2f1b457ab4c9461a4cb3274ea00974746076e2efc8d71f0c1076c0618d> > (134) > - *M* bin/storm-kafka-monitor > <https://github.com/apache/storm/pull/8819/files#diff-a806c19bd872353d150ac6f85e78696285c99021ec81e375cdb904105bf6b16b> > (9) > - *A* bin/storm-kafka-monitor-fetch > <https://github.com/apache/storm/pull/8819/files#diff-3507481db71b11836d4e44a6308c2ace3cc3d545d51368f06efbd0eec6d467c8> > (133) > - *M* bin/storm.py > <https://github.com/apache/storm/pull/8819/files#diff-a2bfdaa7016689362ea4b267e85618a79cdabe435f62e3f40922042ee0eda746> > (11) > - *M* bin/test_storm.py > <https://github.com/apache/storm/pull/8819/files#diff-e34f82e2a1f7d5319006726ad01257d99ec140bd20cbecbf28e90248d8622d1f> > (13) > - *A* external/storm-autocreds/README.md > <https://github.com/apache/storm/pull/8819/files#diff-236662b9419b8399401ec9ad60b84cb7f7c66eca2db8e844018afa55fdd5aa56> > (101) > - *M* external/storm-kafka-monitor/README.md > <https://github.com/apache/storm/pull/8819/files#diff-619cc223ea00cdeadb622a6daa7b21571fe89bf1d634d08f2791dbd2f84422d2> > (24) > - *M* storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java > <https://github.com/apache/storm/pull/8819/files#diff-6a2b2f6b97f1f48abfec324eed8b88442cef3e1e7b85b6a3870557ed9c77f288> > (36) > - *M* storm-dist/binary/final-package/pom.xml > <https://github.com/apache/storm/pull/8819/files#diff-45a6af025058edd4abd6d0d30a89d9bae2ed39a1b04f8dbb45d24a5d6b45188a> > (67) > - *M* storm-dist/binary/final-package/src/main/assembly/binary.xml > <https://github.com/apache/storm/pull/8819/files#diff-ce85984275b435c0dbfa5ea5894afd5617a2bfe3e7a83da24307437f9c5df1ca> > (46) > - *A* storm-dist/binary/final-package/src/main/scripts/dedup-libs.py > <https://github.com/apache/storm/pull/8819/files#diff-2cbebf9b5baea2e59daf389c4e72f59fc37a65c97b1d18e1f6b9f9d9e21e0408> > (104) > - *M* storm-dist/binary/pom.xml > <https://github.com/apache/storm/pull/8819/files#diff-c8dc3f04eecd6165587214ab962cf8684c19f94f2f83cb1392f9ad29c677a18d> > (2) > - *D* storm-dist/binary/storm-autocreds-bin/pom.xml > <https://github.com/apache/storm/pull/8819/files#diff-b113238a66a1c9c9b8bb7c3ad47bdf98207e8d9bebbcea490735bd6c6e9ad8ee> > (63) > - *D* > storm-dist/binary/storm-autocreds-bin/src/main/assembly/storm-autocreds.xml > <https://github.com/apache/storm/pull/8819/files#diff-ecff7ffca0d7bb4801e5d73cb9656abf15934c489e51b97e44e6324da9f5de49> > (33) > - *D* storm-dist/binary/storm-kafka-monitor-bin/pom.xml > <https://github.com/apache/storm/pull/8819/files#diff-07714eeba780f3e65d20c2de3766264ebc8ff2c92be09b47924f21f64f042779> > (69) > - *D* > storm-dist/binary/storm-kafka-monitor-bin/src/main/assembly/storm-kafka-monitor.xml > <https://github.com/apache/storm/pull/8819/files#diff-f07c2d882c56fb747595d07a9eee81e91cd5ee7d9c70d232eaf63eafe6f30567> > (33) > > Patch Links: > > - https://github.com/apache/storm/pull/8819.patch > - https://github.com/apache/storm/pull/8819.diff > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/storm/pull/8819?email_source=notifications&email_token=AAG5GIQLJVAPLMH7D427VH35CP7ZXA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF4ZTSNRVGM4TAMRTGWTHEZLBONXW5MDSMV3GSZLXL5ZGK4LVMVZXIZLEUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAG5GIVLTAJSY645ONYQGKL5CP7ZXAVCNFSNUABEKJSXA33TNF2G64TZHMYTIMJTGU2DOMB3JFZXG5LFHM2DONZZGM2TSNJQHCQXMAQ> > . > You are receiving this because your review was requested.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
