rzo1 opened a new pull request, #8819: URL: https://github.com/apache/storm/pull/8819
## What & why The binary distribution ships ~395 MB of jars, much of it optional or duplicated. This PR trims it substantially without removing any capability: optional pieces become fetch-on-demand, and jars shared between the daemon and worker classpaths are de-duplicated. Smaller artifact means less download/storage/registry bandwidth and a smaller container image, and a lighter CI/CD carbon footprint. 🌱 ## Changes 1. **storm-autocreds no longer bundled (-79 MB).** It pulls the full Hadoop/HBase client tree but is only used on secure (Kerberos) clusters and is off by default. Now ships only the README (like the other `external/*` connectors); `bin/storm-autocreds-fetch` retrieves the plugin and its deps into `extlib-daemon`. 2. **storm-kafka-monitor no longer bundled (-38 MB).** Only needed to show Kafka spout lag in the UI or to run `bin/storm-kafka-monitor`. `bin/storm-kafka-monitor-fetch` installs it into `lib-tools/storm-kafka-monitor`. The UI degrades gracefully when it is absent (`TopologySpoutLag` detects it, shows an actionable message, logs once) and the wrapper prints a hint instead of `ClassNotFound`. 3. **`lib-common` de-duplication (-71 MB).** The worker classpath (`lib-worker`) is a byte-identical subset of the daemon classpath (`lib`). Shared jars are now kept once in `lib-common/` (added to both classpaths via `bin/storm.py`) and removed from `lib`. `dedup-libs.py` only merges byte-identical jars (name and sha-256), so no version is silently merged; tool classpaths are left untouched. Net: roughly -188 MB (~47%) of the bundled jar payload, with no loss of functionality. ## Notes for reviewers Opening as draft to get early feedback on the approach. Still need to verify on a Linux system (full `-Pdist` native distribution build). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
