GitHub user zoomingrocket added a comment to the discussion: Best practice for dynamic 'De-serialize From File' in a Producer/Consumer pattern
@dmainou - Great suggestion, I will explore it further! The current job metrics are just capturing & printing total rows processed vs inserted vs updated vs deleted in log for our stakeholder KPIs. Since we run all pipelines via Airflow + Hop run, we push all of it via Open Telemetry and aggregate the data for KPIs. We did try direct Hop run auto instrumentation using Open Telemetry Java Agent, but since Hop uses custom logging it was did not capture anything, but I will defyexplore dispatching Pipeline metrics to Otel. Thanks! GitHub link: https://github.com/apache/hop/discussions/6545#discussioncomment-15776136 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
