Updates regarding my test with Storm 2.6.0 Snapshot 2023/11/02:
1. I have modified my KO bolts using this pattern in their prepare() method:
long cacheMaxSize =
Long.parseLong(stormConf.get(ConfKeys.SVC_DEF_CACHE_SIZE).toString());
instead of:
long cacheMaxSize = (Long) stormConf.get(ConfKeys.SVC_DEF_CACHE_SIZE);
2. I have found the cause of the regression in loss of 'Storm UI'
title at the top left-hand side of all Storm UI's pages.
This is a mistake of mine in this PR:
https://github.com/apache/storm/pull/3580/commits/0320ee6f2d367086b973b504a976ae6aff853ccd
More precisely, at line 19 of this file:
https://github.com/apache/storm/pull/3580/commits/0320ee6f2d367086b973b504a976ae6aff853ccd#diff-6d3a4162a8af826a25a2b8cd6763c08e9bc5165b736b45890b84fa50597d7481
we must have:
<h1><a href="/">{{title}}</a></h1>
instead of:
<h1><a href="/">{{ui.title}}</a></h1>
I think I missed this during my tests because of browser's cache
(bad excuse, I know...): probably I tested by-patching my deployed
Storm cluster, and I finally forgot to checkout this latest change of
my delivery.
I had to use "private navigation" to make sure this time it fixes this issue
What would be the quickest way for this very simple fix to be delivered ?
Thanks,
Alexandre
Le ven. 3 nov. 2023 à 20:57, Alexandre Vermeerbergen
<avermeerber...@gmail.com> a écrit :
>
> Thanks Richard for the analysis.
>
> My thought on this are very simple: let me fix my own old code, after
> all this was impacting only one of my topologies, the other one were
> using more careful Integer.parseInt or Integer.parseLong to avoid
> making assumption on the way the values in the config Map were
> serialized.
>
> My other concern with 2.6.0 Snapshot is that in Storm UI, the "Storm
> UI" title at the top left hand side of each page is missing. This may
> be the result of one of my own PR, I'll have to check why...
>
> Alex
>
> Le ven. 3 nov. 2023 à 20:23, Richard Zowalla <r...@apache.org> a écrit :
> >
> > Hi,
> >
> > thanks for the feedback.
> >
> > The main difference to 2.5.0 is, that we switched the JSON
> > implementation in Utils, which relied on a
> > super old JSON library. What is happening here is, that the new JSON
> > library (net.minidev) behaves differently.
> >
> > We can see it by writing some simple code without Storm (The related
> > code line with the parsing is taken from Storm Utils)
> >
> >
> > Map<String, Long> conf = new HashMap<>();
> > conf.put("test", 100000000L);
> >
> > Map<String, Object> deserTopoConf = (Map<String, Object>)
> > JSONValue.parseWithException(JSONValue.toJSONString(conf));
> >
> > for(Map.Entry<String, Object> e : deserTopoConf.entrySet()) {
> > System.out.println(e.getValue().getClass().getSimpleName());
> > }
> >
> > This will print "Integer". Using "10000000000000000L" as a value will
> > result in "Long".
> >
> > The main issue derives from the fact, that we serialize the config map
> > into a JSON String and parse the result.
> > Based on the String representation, "100000000L" will still fit into an
> > Integer, so the parser will not use Long ;-)
> >
> > If the number is big enough, the parser will switch to a Long.
> > The old library (> 10 years old now):
> >
> > deserTopoConf = (Map<String, Object>)
> > org.json.simple.JSONValue.parseWithException(org.json.simple.JSONValue.
> > toJSONString(conf));
> >
> > for(Map.Entry<String, Object> e : deserTopoConf.entrySet()) {
> > System.out.println(e.getValue().getClass().getSimpleName());
> > }
> >
> > This will print "Long".
> >
> > The difference between these libraries is, that the super old
> > org.json.simple just always returns Long as the type.
> >
> > If can eventually tune the new JSON parser to something like:
> >
> > deserTopoConf = (Map<String, Object>) new
> > JSONParser(JSONParser.BIG_DIGIT_UNRESTRICTED).parse(JSONValue.toJSONStr
> > ing(conf), JSONValue.defaultReader.DEFAULT);
> >
> > This will result in "Long" but might have other side-effects.
> >
> > From my point of view, the newer JSON library works as expected
> > (although it might break backward compatibility).
> > Unchecked casts are always evil, imho. In your specific case, it might
> > be more save to cast to "Number" and than invoke "longValue()" instead.
> >
> > Nevertheless, we might being able to restore the old behaviour by
> > tuning the JSONParser configuration.
> >
> > Thoughts?
> >
> > Gruß
> > Richard
> >
> >
> >
> >
> > Am Freitag, dem 03.11.2023 um 18:32 +0100 schrieb Alexandre
> > Vermeerbergen:
> > > On a side note, the Long value that "MyKOBolt" is set by a
> > > DefaultTopologyConfig to a value with is a java.lang.Long instance...
> > > hence it's surprising to see it internally converted into a
> > > java.lang.Integer in the Map passed to prepare() method...
> > >
> > >
> > > Le ven. 3 nov. 2023 à 18:27, Alexandre Vermeerbergen
> > > <avermeerber...@gmail.com> a écrit :
> > > >
> > > > Hello,
> > > >
> > > > I found a breaking change using Apache Storm 2.6.0 snapshot build
> > > > dated 20231103 and without changing my topologies' code (keeping
> > > > the
> > > > same big jars).
> > > >
> > > > This seems to be tied to a datatype change in Map argument of
> > > > prepare() method of Bolt.
> > > >
> > > > For example, this Bolt is KO at runtime:
> > > >
> > > > public class MyKOBolt extends BaseRichBolt {
> > > >
> > > > @Override
> > > > public void prepare(Map stormConf, TopologyContext context,
> > > > OutputCollector collector) {
> > > > cacheMaxSize = (Long)
> > > > stormConf.get(ConfKeys.SVC_DEF_CACHE_SIZE);
> > > > cacheTtlMin = (Long)
> > > > stormConf.get(ConfKeys.SVC_DEF_CACHE_TTL_MIN);
> > > >
> > > > as it throws this error:
> > > > 2023-11-03 16:42:20.230 o.a.s.e.b.BoltExecutor
> > > > Thread-14-__system-executor[-1, -1] [INFO] Preparing bolt
> > > > __system:[-1]
> > > > 2023-11-03 16:42:20.193 o.a.s.u.Utils
> > > > Thread-15-evaluateTriggers-executor[11, 11] [ERROR] Async loop
> > > > died!
> > > > java.lang.ClassCastException: java.lang.Integer incompatible with
> > > > java.lang.Long
> > > > at com.acme.storm.alerting.MyKOBolt
> > > > .prepare(EvaluationBolt.java:71)
> > > > ~[stormjar.jar:?]
> > > > at
> > > > org.apache.storm.executor.bolt.BoltExecutor.init(BoltExecutor.java:
> > > > 128)
> > > > ~[storm-client-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
> > > > at
> > > > org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java:
> > > > 138)
> > > > ~[storm-client-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
> > > > at
> > > > org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java:
> > > > 54)
> > > > ~[storm-client-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
> > > > at org.apache.storm.utils.Utils$1.run(Utils.java:393)
> > > > [storm-client-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
> > > > at java.lang.Thread.run(Thread.java:857) [?:?]
> > > >
> > > > whereas this bolt is fine with Storm 2.6.0:
> > > >
> > > > public class MyOKBolt extends BaseBasicBolt {
> > > >
> > > > @Override
> > > > public void prepare(final Map stormConf, final TopologyContext
> > > > context) {
> > > > super.prepare(stormConf, context);
> > > >
> > > > this.redisPort =
> > > > Integer.parseInt((String)
> > > > stormConf.get(ConfKeys.REDIS_PORT));
> > > >
> > > > That said, I can modify the code of "MyKOBolt" to use same
> > > > Integer.parseInt or Integer.parseLong trick, but it's the first
> > > > time
> > > > in my long history of upgrades that I have seen such runtime
> > > > incompatibility.
> > > >
> > > > Maybe "MyKOBolt" was badly written since the beginning and I have
> > > > just
> > > > hit the punishment for it: is there a documentation which clarifies
> > > > datatypes of the Map argument of prepare() method?
> > > >
> > > > Also worth noting: "MyKOBolt" derives from BaseRichBolt , while
> > > > "MyOKBolt" derives from BaseBasicBolt => could this have any impact
> > > > on
> > > > this finding ?
> > > >
> > > > Last, I'm running this on Redhat Linux 8 and IBM Semeru JDK
> > > > 17.0.8.1.
> > > >
> > > > Kind regards,
> > > > Alexandre
> > > >
> > > > Le ven. 3 nov. 2023 à 15:16, Julien Nioche
> > > > <lists.digitalpeb...@gmail.com> a écrit :
> > > > >
> > > > > Thanks Richard.
> > > > >
> > > > > Tried the latest snapshot with StormCrawler both in local and
> > > > > deployed mode
> > > > > and did not find any issues.
> > > > > Will try it on a topology generating WARC files next week to
> > > > > check that the
> > > > > dependency changes on Hadoop have not broken anything.
> > > > >
> > > > > Have a good week end
> > > > >
> > > > > Julien
> > > > >
> > > > > On Thu, 2 Nov 2023 at 19:25, Richard Zowalla <r...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Short update. Sorted out an issue with the tar.gz/zip with
> > > > > > Julien today
> > > > > > and re-uploaded them to the nightlies area.
> > > > > >
> > > > > > This new bundle works as expected in my deployment but happy to
> > > > > > receive
> > > > > > additional feedback before getting up a first release candidate
> > > > > > ;-)
> > > > > >
> > > > > > Am Montag, dem 30.10.2023 um 08:21 +0100 schrieb Richard
> > > > > > Zowalla:
> > > > > > > Hi Alexandre,
> > > > > > >
> > > > > > > we are not in a hurry here :) - take as much as time you
> > > > > > > need.
> > > > > > >
> > > > > > > Gruß
> > > > > > > Richard
> > > > > > >
> > > > > > >
> > > > > > > Am Montag, dem 30.10.2023 um 07:50 +0100 schrieb Alexandre
> > > > > > > Vermeerbergen:
> > > > > > > > Hello Richard,
> > > > > > > >
> > > > > > > > Okay, I'm more than happy to do that with my pre-production
> > > > > > > > cluster
> > > > > > > > (~10 topologies) using the binary artifacts.
> > > > > > > > Would it be OK if I can use up to end of this week so that
> > > > > > > > I'll be
> > > > > > > > able to have enough time to check all potential issues that
> > > > > > > > this
> > > > > > > > upgrade bring?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Alexandre
> > > > > > > >
> > > > > > > > Le lun. 30 oct. 2023 à 07:35, Richard Zowalla
> > > > > > > > <r...@apache.org> a
> > > > > > > > écrit :
> > > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > before starting a release of 2.6.0-SNAPSHOT, I would like
> > > > > > > > > to
> > > > > > > > > receive
> > > > > > > > > some feedback on the current SNAPSHOT build.
> > > > > > > > >
> > > > > > > > > I just uploaded a 2.6.0-SNAPSHOT of Storm build from
> > > > > > > > >
> > > > > > https://github.com/apache/storm/commit/8f883086032669a8f04b09a3b312d60af5b44533
> > > > > > > > >
> > > > > > > > > It is available via the ASF Snapshot repository:
> > > > > > > > > https://repository.apache.org/content/repositories/snapshots/
> > > > > > > > >
> > > > > > > > > You can consume by adding
> > > > > > > > >
> > > > > > > > > <repository>
> > > > > > > > > <id>apache.snapshots</id>
> > > > > > > > > <name>Apache Snapshot Repository</name>
> > > > > > > > > <url>https://repository.apache.org/snapshots</url>
> > > > > > > > > <releases>
> > > > > > > > > <enabled>false</enabled>
> > > > > > > > > </releases>
> > > > > > > > > </repository>
> > > > > > > > >
> > > > > > > > > to your project pom. As we do not deploy SNAPSHOTS
> > > > > > > > > automatically,
> > > > > > > > > it
> > > > > > > > > should be easy to just consume the latest SNAPSHOT.
> > > > > > > > >
> > > > > > > > > The packaged binaries are available at
> > > > > > > > > nightlies.apache.org:
> > > > > > > > >
> > > > > > > > >
> > > > > > https://nightlies.apache.org/storm/2.6.0-SNAPSHOT/8f883086032669a8f04b09a3b312d60af5b44533/
> > > > > > > > >
> > > > > > > > > If you have some minutes left: Please test and report any
> > > > > > > > > issues
> > > > > > > > > with
> > > > > > > > > this binaries, so we can fix before attempting to
> > > > > > > > > release.
> > > > > > > > >
> > > > > > > > > The most significant changes are in the are of
> > > > > > > > > hadoop/hbase/hdfs
> > > > > > > > > as
> > > > > > > > > we
> > > > > > > > > upgraded from 2.x to 3.x - our own test coverage within
> > > > > > > > > the build
> > > > > > > > > looks
> > > > > > > > > good but would be nice to get some real world use-case
> > > > > > > > > feedback
> > > > > > > > > on
> > > > > > > > > this. In addition, we had quite a lof of 3rd party
> > > > > > > > > dependency
> > > > > > > > > upgrades.
> > > > > > > > >
> > > > > > > > > In addition, it contains the pruning of external modules
> > > > > > > > > as
> > > > > > > > > listed
> > > > > > > > > in
> > > > > > > > >
> > > > > > > > > https://issues.apache.org/jira/browse/STORM-3988
> > > > > > > > >
> > > > > > > > > A summaryin Jira is here:
> > > > > > > > >
> > > > > > > > >
> > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314820&version=12353484
> > > > > > > > >
> > > > > > > > > Gruß
> > > > > > > > > Richard
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > *Open Source Solutions for Text Engineering*
> > > > >
> > > > > http://www.digitalpebble.com
> > > > > http://digitalpebble.blogspot.com/
> > > > > #digitalpebble <http://twitter.com/digitalpebble>
> >
