My main concerns are: - Performance: adding metrics should have (close to) no impact on current performance. For short builds metrics might not add enough value, and our aim should be fast and short builds. Metrics is a tool to analyze in case you seek improvements of your builds - Portability:Coincidentally there was just a thread about logging as well, Plexus Logger versus SLF4J. One of the things you see is that we regret that Plexus Logger is part of the API. In Maven core there is no logging API, so for the same reason I think there should be no metrics API in Maven core either. Portability is about users being able to pick there own tool to do analysis Portability is about plugin-developers users being able to pick there own metrics API if they want.
Portability is about Maven being able to switch to a future standard, like micrometer, microprofile or something else. Robert On 1-6-2020 07:46:16, Enrico Olivelli <eolive...@gmail.com> wrote: Il Lun 1 Giu 2020, 07:08 Romain Manni-Bucau ha scritto: > Hi Enrico, > > As mentionned I think 5 would be an error, we should provide an in memory > flavor with a dump at exist (-Dmaven.metrics.dumpOnExit=/tmp/metrics.csv or > so?) otherwise it will be a noop for user so not sure it is worth having it > at all. > Here it is the implementation It is only a matter of pushing it to our ASF repository. It is mostly a cut and paste of the same stuff we did in Zookeeper project. No problem in pushing it to Maven repository from my side. Enrico In other words, if you want it as a vendor you put a javaagent on the JVM > and you are done with the added value of getting metrics. > I don't mean I think we must not have it, I strongly think we must, but I > also think it is only worth if usable directly, even if limited to in > memory case. > > Romain Manni-Bucau > @rmannibucau | Blog > | Old Blog > | Github <> > https://github.com/rmannibucau> | > LinkedIn | Book > <> > https://www.packtpub.com/application-development/java-ee-8-high-performance > > > > > Le dim. 31 mai 2020 à 22:31, Enrico Olivelli a > écrit : > > > Hey guys, > > I feel this Metrics Runtime API will be very useful to Maven and Maven > > plugins. > > But I am not sure we are reaching consensus. > > > > Main points: > > 1) We will introduce a Metrics Provider API, maintained by Maven project. > > 2) Metrics are not "extension points", they are not "events", > > implementations of the Provider API cannot alter the behaviour of Maven > > 3) Metrics are just "values in time" (counters, gauges, summaries) , > > usually exposed as time series, they are useful to inspect the runtime > > low-level performances of internals of Maven core, internal libraries and > > plugins > > 4) The implementation of the API (the code that publishes and/or analyse > > data) is loaded using the Extensions mechanism, by default there will be > a > > noop implementation bundled with Maven distribution > > 5) We (Maven project) won't provide initially any implementation other > than > > the NOOP, in order not to add burden to Maven project. I can maintain a > few > > implementations on a separate repository, no need to add a burden to main > > Maven project. > > 6) Plugins will be able to register metric values using the Metrics API, > > the will look up the MetricSystem component with @Inject or with explicit > > lookup. > > > > I am glad to move forward with this project if we agree on all of the > > aspects. > > I am open to discuss again about all of the points. > > > > Thanks to everyone who already joined the discussion, I hope we could > ship > > v1 of the API in Maven 3.7 or Maven 3.8, the sooner the better > > > > Enrico > > > > > > > > Il giorno lun 11 mag 2020 alle ore 08:02 Romain Manni-Bucau <> > > rmannibu...@gmail.com> ha scritto: > > > > > Hmm, it shouldn't be: > > > > > > metricsSystem > > > .getMetricsContext() > > > .getSummary( "resolvePluginDependency", "Time to resolve dependencies > of > > a > > > plugin (ms)" ) > > > .add( MetricsUtils.elapsedMillis( startResolve ) ); > > > > > > but > > > > > > counter.add(duration). > > > > > > Resolution of the counter can be either (just in terms of inputs, then > > api > > > can be fluent or not, it is a detail for me): > > > > > > Counter counter = metricSystem.get(new CounterSpec("my counter", > "ms")); > > > > > > or, since any operation in maven has a scope (mojo, artifact > resolution, > > > ...): > > > > > > @Inject @MetricSpec("my-counter", "ms") Counter counter; > > > > > > Concretely metrics system should enable to resolve a counter from a few > > > meta (at least name, likely also the unit) and counter is just a long > > (then > > > in terms of impl it is a LongAdder to be concurrency friendly I guess > but > > > it is a detail). > > > We likely don't want to pay any other overhead otherwise I fully agree > > > events are almost 1-1 in terms of feature but totally opposed in terms > of > > > design: > > > > > > 1. A counter is a unified view of data where data are pushed from > > > contributors > > > 2. Event are an heterogeneous set of data where consumers are > > interpreting > > > the value > > > > > > If you want to be iso you create a Counter event but then you have this > > > overhead we should avoid IMO + you just recreated metric system with > > > another API (likely slower due to the bus usage which requires a lot of > > > caching and JIT to be iso in terms of perf but it is really after some > > > dozen of thousands of executions). > > > > > > Romain Manni-Bucau > > > @rmannibucau | Blog > > > | Old Blog > > > | Github <> > > > https://github.com/rmannibucau> | > > > LinkedIn | Book > > > <> > > > > > > https://www.packtpub.com/application-development/java-ee-8-high-performance > > > > > > > > > > > > > Le dim. 10 mai 2020 à 22:23, Enrico Olivelli a > > > écrit : > > > > > > > Il giorno dom 10 mag 2020 alle ore 22:10 Robert Scholte <> > > > > rfscho...@apache.org> ha scritto: > > > > > > > > > This looks more like a bug, so this should be analyzed. > > > > > It sounds to me, that if this information was accurate, there > > wouldn't > > > be > > > > > a need for the a separate MetricSystem. > > > > > > > > > > > > > Sorry, I have added that metric just to have some data to retrieve > and > > > > monitor. > > > > Maybe this is not interesting. > > > > > > > > The point is that if I want to track this duration, now I can do it > > just > > > by > > > > adding a bunch of lines locally. > > > > There is no need to introduce a new formal "event", that would be > part > > of > > > > some public API. > > > > If the information is not useful I can remove it in the next release, > > > there > > > > is no strong contract with the end user > > > > > > > > Enrico > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Robert > > > > > On 10-5-2020 20:38:39, Romain Manni-Bucau > > > wrote: > > > > > Le dim. 10 mai 2020 à 20:11, Karl Heinz Marbaise a > > > > > écrit : > > > > > > > > > > > > > > > > > On 10.05.20 19:59, Romain Manni-Bucau wrote: > > > > > > > Events vs metrics is an old topic. > > > > > > > The choice between both is not only design but also about perf. > > In > > > > > terms > > > > > > of > > > > > > > design it brings the ability to export data without exposing it > > in > > > > code > > > > > > > (events are public). It also avoid to expose a stable api of > > events > > > > and > > > > > > > create coupling between plugin/exts and only require a single > > > stable > > > > > api > > > > > > > which is very minimal (lot of projects kept their metric > > > abstraction > > > > > > > stable). > > > > > > > > > > > > > > Typically wagon should export metrics but if you fire an event > > per > > > > > > download > > > > > > > progress it will be quite slow currently and you will allocate > a > > > lot > > > > of > > > > > > > objects (or create contentions) just to increase a monotonic > > > counter > > > > (a > > > > > > > longadder concretely). > > > > > > > > > > > > > > So I think Enrico is right and we cant avoid metrics (I have to > > > admit > > > > > > > events are overcomplex to use for plugin/ext dev, see all > > profiling > > > > > > > extensions, > > > > > > > > > > > > > not a single one works and report accurate data whereas metrics > > > > > > > can miss some drilldown data but would be right). > > > > > > > > > > > > Interesting point cause can you give some more details on which > the > > > > > > testimony is based that none of them works? > > > > > > > > > > > > > > > > They all measure bus events, mainly mojo_start, mojo_end but at the > > end > > > > > several mojo have a duration of 0ms...even if they take minutes. > > > > > > > > > > What I saw last time I checked the two top rated by github (and > > google > > > I > > > > > think), some events were missing (one ex where metrics are easier > to > > > > > handle, you push your data, no computation on consumer side). > > > > > Out of my head in the project i was working on, gmaven plugin was > > > totally > > > > > missed. > > > > > > > > > > > > > > > > Can you explain a accurate implementation? > > > > > > > > > > > > > > > > If my build take 5mn and i sum all not overlapping times of the > > report > > > > of a > > > > > monothreaded (T1) build then i should be close to 5mn (+- some sec > > for > > > > the > > > > > boot, ioc, logging, final report). > > > > > If I cant extract where time is spent these tools are uselesd. > > > > > > > > > > For the story i used a javaagent (based on sirona) to instrument > the > > > > build > > > > > i was optimizing, was using in memory counters with dump at > shutdown > > > (in > > > > > csv). Indeed bytecode instrumentation overhead was not low enough > > but i > > > > > managed to explain the build duration properly with such a tool. > > > > > > > > > > If we put these basic metrics in maven core (executor for ex) I'm > > > pretty > > > > > confident results will be less random than events interpretation > and > > > > > required code is very low and trivial to maintain for any dev so > > guess > > > it > > > > > is worth it even if off by default (noop metrics system). > > > > > > > > > > > > > > > > Kind regards > > > > > > Karl Heinz Marbaise > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Now we can limit the surfacing api and start only by counters > and > > > > > gauges > > > > > > > maybe and later add histogram, meters etc only if needed (not > > sure > > > > for > > > > > > mvn). > > > > > > > > > > > > > > Le dim. 10 mai 2020 à 19:32, Enrico Olivelli a > > > > > > écrit : > > > > > > > > > > > > > >> Il Dom 10 Mag 2020, 17:49 Robert Scholte ha > > > > > > >> scritto: > > > > > > >> > > > > > > >>> Hi Enrico, > > > > > > >>> > > > > > > >>> These could all be solved by firing events. > > > > > > >>> > > > > > > >> > > > > > > >> Yes > > > > > > >> I see your point. > > > > > > >> Unfortunately those are the first points that I have > > instrumented > > > in > > > > > > order > > > > > > >> to see some data. > > > > > > >> > > > > > > >> > > > > > > >>> And there's another benefit: since there won't be an API, we > > > don't > > > > > need > > > > > > >> to > > > > > > >>> maintain it,or be afraid of mistakes. > > > > > > >>> > > > > > > >> > > > > > > >> Exposed metrics won't be an API, or at least something to be > > > > > maintained. > > > > > > >> Each Maven version, component version, plugin version may > expose > > > any > > > > > > metric > > > > > > >> is valuable for the component developer or for the users. > > > > > > >> > > > > > > >> Exposed metrics are not an extension point, it is not expected > > > that > > > > a > > > > > > >> MetricsProvider is able to alter the behaviour of Maven. > > > > > > >> Events/Extension point must be maintained forever, but you can > > > > > drop/add > > > > > > a > > > > > > >> metric at each release. They are very low level tracking > points > > in > > > > > code. > > > > > > >> > > > > > > >> This is because I say that metrics are like loggers: you add > > logs > > > > > where > > > > > > you > > > > > > >> think it is useful, you can change log point at every version, > > > there > > > > > is > > > > > > not > > > > > > >> strong contract with the user, logging implementation is > totally > > > > > > pluggable. > > > > > > >> > > > > > > >> The Metrics API is mostly the contract with Metrics Provider > > > > > developers. > > > > > > >> It will be very stable, I don't think there will be many > changes > > > in > > > > > the > > > > > > mid > > > > > > >> term. > > > > > > >> It is only about the ability to give a name to counters, > > summaries > > > > and > > > > > > >> gauges. > > > > > > >> > > > > > > >> Having a simple implementation that dumps the final values at > > the > > > > end > > > > > is > > > > > > >> not the full power of this mechanism. > > > > > > >> > > > > > > >> You can attach Prometheus or any other time series based > metrics > > > > > > database > > > > > > >> and see the evolution of the build, this will be very useful > for > > > > long > > > > > > >> running builds and especially on CI or during component > > > development. > > > > > > >> > > > > > > >> I will finish the implementation with Prometheus and try to > > > create a > > > > > > >> Grafana dashboard. > > > > > > >> > > > > > > >> It would be super useful that some Maven-internals experts > could > > > > > suggest > > > > > > >> useful values to track during the execution of the build. > > > > > > >> > > > > > > >> I hope that explains better the value of the Metrics system. > > > > > > >> > > > > > > >> Thank you Robert for your feedback and help > > > > > > >> > > > > > > >> Enrico > > > > > > >> > > > > > > >> > > > > > > >> To me this is the wrong approach to embed metrics. > > > > > > >>> > > > > > > 7 > > > > > > > > > > > > > > > > > > > > >
