Thanks for the reply Aaron, Rui Why do you need to update dependencies on external-lib on both Nimbus > and Supervisor?
Like we may use jackson, kafka client, apache commons, other internal frameworks jars and other third parties jars, so we do have those in the external-lib, on the jar version changes we do change those jars in external-lib. Are you not submitting an uber-jar with all your dependencies plus > your topology? If I add all dependencies in the topology jar it's size is too large around 350MB. Announce deployment well ahead of time - allowing users to plan switches to > other clusters if desired. Aaron, i'm asking for the topology code changes and internal jar version changes update. Not for upgrading the Storm version, sorry for not setting proper questions here. On Fri, Sep 20, 2024 at 8:57 PM Rui Abreu <rui.ab...@gmail.com> wrote: > Hello Karthick, > Why do you need to update dependencies on external-lib on both Nimbus > and Supervisor? > Are you not submitting an uber-jar with all your dependencies plus > your topology? > > > > On Fri, 20 Sept 2024 at 08:10, Karthick <ibmkarthickma...@gmail.com> > wrote: > > > > Dear Apache Storm Community, > > > > I am currently managing an Apache Storm cluster with 38 nodes: 3 > dedicated > > to ZooKeeper, 1 to Nimbus and the UI, and 34 nodes running Supervisor and > > Logviewer processes. Each node has 2 Workers. > > > > At present, our topology update process involves the following steps: > > > > 1. Killing the existing topology. > > 2. Changing dependency JARs under the external-lib dir and restarting > > Nimbus. > > 3. Changing dependency JARs under the external-lib dir and restarting > > Supervisors. > > 4. Submitting the new topology. > > > > Each operation takes about 2–3 minutes. As the number of Supervisor nodes > > increases, the overall time for topology updates is becoming a concern. > > > > I am reaching out to seek advice on how to optimize this process, as I > > believe there are more efficient ways to handle topology updates in > > large-scale Storm deployments. Specifically: > > > > - Is there a more efficient process to handle code changes without > > having to manually restart Nimbus and Supervisors? > > - How can I reduce the overall time for topology updates, especially > as > > our cluster continues to grow? > > - Are there industry-standard practices for implementing rolling > updates > > or automating the deployment process? > > > > Any insights, recommendations, or best practices that could help > streamline > > our update process would be greatly appreciated. > > > > Thank you for your time, and I look forward to your suggestions! >
