rzo1 commented on PR #8819: URL: https://github.com/apache/storm/pull/8819#issuecomment-4858636779
Thanks @reiabreu and @GGraziadei - keeping the air-gapped/offline case working is exactly the goal. **The fetch scripts already support internal mirrors**, because they delegate to Maven instead of hard-coding Maven Central, so Maven honors your existing mirror/proxy setup: - A host whose `~/.m2/settings.xml` (or `$MAVEN_HOME/conf/settings.xml`) has a `<mirror>` with `<mirrorOf>*</mirrorOf>` pointing at your Nexus/Artifactory works with **no extra flags**. - Explicit file: `bin/storm-kafka-monitor-fetch -- -s /etc/maven/settings.xml` - Fully offline against a pre-seeded local repo: `... -- -Dmaven.repo.local=/srv/offline-repo -o` I went with Maven resolution over a raw download precisely for this: enterprises already have mirror + proxy + auth in settings.xml, so we reuse all of it (credentials, `mirrorOf` patterns) instead of reinventing it. **The key point: fetch is a provisioning-time step, not a per-node runtime requirement - and it fits how operators already work.** In hardened environments you *don't* want production nodes downloading anything; artifacts flow through a controlled prep/staging pipeline. That's the model here: run fetch once on an admin/build/CI box with mirror access, prep for the target environment, and copy the jars into `extlib-daemon` / `lib-tools/...` (or bake them into the image). The locked-down nodes stay as sealed as they are today. autocreds is the clearest case: turning it on is *already* environment-specific prep - you stage the cluster's `hdfs-site.xml`/`hbase-site.xml`, keytabs, and wire the principals + plugin classes into `storm.yaml`. Copying the Hadoop client jars is just one more step in a task the operator is doing anyway, so bundling ~79 MB of Hadoop into *every* download to save that one copy is a poor trade. **And where a batteries-included artifact really is wanted, we can express that as Docker image variants rather than a second source release** - e.g. a lean `apache/storm:<ver>` plus a `apache/storm:<ver>-full` (Hadoop/Kafka libs pre-fetched). Enterprise users who need the libs present just pull the full tag - no custom rebuild - while the source release stays single and lean. That moves the full-vs-slim split into our image CI (cheap, already automated) and off the Apache release process (no extra signing/vote/artifacts), and it sidesteps the "everyone grabs full" concern since the default/base image stays slim. For boxes with no system Maven, `mvnw` works too - we can auto-detect one, its distribution coming from the same internal mirror. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
