rmannibucau commented on issue #8321: URL: https://github.com/apache/hop/issues/8321#issuecomment-5663188136
> run state, metrics, parent/child hierarchy, data profiles, “give me the complete picture of this run when it finishes.” That is a Hop feature on purpose. Spark/Flink/Beam are weak at this, and it is one of the reasons people run Hop on those engines instead of only using the engine UI. Hmm, I don't undersand this beam and spark are quite good at that actually 🤔 , indeed it needs some setup - but it applies to hop depending your prod stack as well. That said there might be something ambiguous in the way you summarize it - or not obvious in my PR, pick the right one ;): * metrics are intended to be a specific thing * tracing is also intended to be specific these two are were hop without the UI/server can be weaker than spark runtime where otel is more common but this is also the sense to move to a more standard layer for the first of the 4 observability pillars (the logging) - note that I skip profiling for now, this is wayyyy lesssss critical on the next step I'll try to address your issues targetting next minor - measured locally the cost of plain logging (understand this is not significative as soon as you have any transform even the row generator one) and cost is more or less the one your measured when everything is enabled including method/line capture at runtime (but we still speak of us) I'm still not convinced by logchannel abstraction - it is not used everywhere (this buffer thing) and defeats the unified logging abstraction, can need more time but it might also mean the API is just wrong and it needs to be split in multiple concerns In terms of plan 1 sounds like an agreement of everybody, think we can refine if we want to just drop everything for 2+3 and move to otel as being centric maybe? (thinking out loud but if that's the final stage no need to invest much in the logging in between maybe). Can mean the UI and server are updated to query a collector but this would be a win overall so worth thinking. So long story short: will open a PR for 1, will try to update this one with the few step were we agree to not keep it in a dirty stage and will keep disagreement there until we discuss further. hope it makes sense -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
