Re: [I] [Feature Request]: Add the "Output steps metrics" transform to HOP (hop)
mattcasters commented on issue #3003: URL: https://github.com/apache/hop/issues/3003#issuecomment-5025827553 Created by my friend Samatar, that's why it escaped my attention: https://github.com/pentaho/pentaho-kettle/blob/master/engine/src/main/java/org/pentaho/di/trans/steps/stepsmetrics/StepsMetrics.java -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Add the "Output steps metrics" transform to HOP (hop)
CarlosJuncher03 commented on issue #3003: URL: https://github.com/apache/hop/issues/3003#issuecomment-4416754342 Of course, this is just my perspective based on some use cases we've seen. I understand that there might be better ways to solve this within Hop's current architecture, but I believe something along these lines could add value. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Add the "Output steps metrics" transform to HOP (hop)
CarlosJuncher03 commented on issue #3003: URL: https://github.com/apache/hop/issues/3003#issuecomment-4416749624 I agree that Pipeline Logging, Pipeline Probe, and Execution Information are excellent ways to monitor the full pipeline flow in a generic and reusable way. In fact, we have been able to create a monitoring model that can be forked and applied to multiple pipelines, which is very powerful. My point is not to replace that approach. The idea is to have something lighter for smaller or more niche pipelines, where setting up the full logging/probe/execution information structure may be more than what is needed. For those cases, it would be useful to have a simple way to expose selected step metrics directly in the standard pipeline log, or through a lightweight transform. This would make basic operational visibility easier without competing with the more complete monitoring approach. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Add the "Output steps metrics" transform to HOP (hop)
CarlosJuncher03 commented on issue #3003: URL: https://github.com/apache/hop/issues/3003#issuecomment-4416739293 I agree that Pipeline Logging is the most suitable generic solution, and we already use Pipeline Logging along with Pipeline Probe for monitoring and controlling the data path. The use case I see here is different: small projects or restricted environments where the standard pipeline log is the only practical place to expose metrics. For example, a simple pipeline running on third-party infrastructure, with limited server access, where configuring a full logging pipeline or centralized monitoring is impractical. In this situation, having a lightweight way to write selected metrics from the steps directly to the standard log would be useful. The goal wouldn't be to replace Pipeline Logging, but to make simple cases easier and more visible. Perhaps this could be implemented as an improvement to Write to Log, or as a simplified way to record selected metrics without creating a separate logging pipeline. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Add the "Output steps metrics" transform to HOP (hop)
mattcasters commented on issue #3003: URL: https://github.com/apache/hop/issues/3003#issuecomment-4414403740 It would be better to explain what you exactly need as I have no idea anymore about what the old step exactly did. Do you wonderful folks really think it's a good idea to manually add a bunch of extra transforms to each pipeline despite Pipeline Logging offering the exact same information in a generic way? Simply go to the metadata perspective, create a new Pipeline Logging element, create a new logging pipeline and you'll get a generic solution that works on your whole project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Add the "Output steps metrics" transform to HOP (hop)
CarlosJuncher03 commented on issue #3003: URL: https://github.com/apache/hop/issues/3003#issuecomment-4409313787 Regarding this request, do you plan to include it? I think it would be good, as it would make it easier to add information about pipeline totals using the write-to-log feature. The write-to-log feature makes it easy to put important information into the standard pipeline log in a simple way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Add the "Output steps metrics" transform to HOP (hop)
krishnasdfw commented on issue #3003: URL: https://github.com/apache/hop/issues/3003#issuecomment-2486982528 I reviewed Exec Info Transform. It seems to read from audit location, but not clear whether that can be directly tied to a current running transform and it's data. Pipeline Log / Pipeline Probe combination seems like an option, but can both the log and probe be setup in same pipeline and both meta data types configured with same pipeline? Will that be a single execution of the pipeline feeding into both inputs in same pipeline execution? This is a valid scenario if the input is just one row of large json message that may get split rows processed during pipeline execution. Ultimately the error handler that examines error hop step metrics , may need to get this first input message and take some action based on error metrics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Add the "Output steps metrics" transform to HOP (hop)
krishnasdfw commented on issue #3003: URL: https://github.com/apache/hop/issues/3003#issuecomment-2480126306 This is one of the great features of PDI. Several simplifications were implemented using this. Getting metrics is not merely informational, but critical to embedded transforms, that process many rows in a run. And then a OSM monitor step , standalone is able to perform error handling and detect if there were traffic to any of the error hops from various steps which may simply log the error. Else there will be too many cross connects, blocks, aggregations using merge etc. to achieve the samething. Without this Hop will lose it's power immensely. We are not processing data any more in files and records, rather JSON messages with nested data collections with PDI and it works!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Add the "Output steps metrics" transform to HOP (hop)
mattcasters commented on issue #3003: URL: https://github.com/apache/hop/issues/3003#issuecomment-2009135516 Google translate says: "Or do you want to implement the same function as the conversion step statistics component? At present, hop has not seen any components with similar functions, which can detect the insertion, update, reading, etc. of insert and update components." and "There is no conversion step statistics for this component! !" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Add the "Output steps metrics" transform to HOP (hop)
openinbigdata commented on issue #3003: URL: https://github.com/apache/hop/issues/3003#issuecomment-1852315272 > > > Workaround is to get this information after the execution with the "Execution Information" transform in another pipeline. > > > > > > Matt, but is reading the "Execution Information" information already possible in 2.7? > > Yes, this is already possible. There's a whole perspective to explore prior executions, but this information is also available through the transform. Matt, do you have any example pipelines using the Execution information transform that you can send me? I confess that I can't use it, as I don't understand how it should be used, so an example would be of great help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Add the "Output steps metrics" transform to HOP (hop)
openinbigdata commented on issue #3003: URL: https://github.com/apache/hop/issues/3003#issuecomment-1851985656 It would be fantastic to have this functionality, just yesterday I needed it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Feature Request]: Add the "Output steps metrics" transform to HOP (hop)
mattcasters commented on issue #3003: URL: https://github.com/apache/hop/issues/3003#issuecomment-1813249343 Workaround is to get this information after the execution with the "Execution Information" transform in another pipeline. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
