[ https://issues.apache.org/jira/browse/GOBBLIN-2179?focusedWorklogId=947719&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-947719 ]
ASF GitHub Bot logged work on GOBBLIN-2179: ------------------------------------------- Author: ASF GitHub Bot Created on: 11/Dec/24 05:16 Start Date: 11/Dec/24 05:16 Worklog Time Spent: 10m Work Description: phet merged PR #4082: URL: https://github.com/apache/gobblin/pull/4082 Issue Time Tracking ------------------- Worklog Id: (was: 947719) Remaining Estimate: 0h Time Spent: 10m > Provide observability about how much work a Gobblin-on-Temporal job will (and > already has) performed > ---------------------------------------------------------------------------------------------------- > > Key: GOBBLIN-2179 > URL: https://issues.apache.org/jira/browse/GOBBLIN-2179 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-core > Reporter: Kip Kohn > Assignee: Abhishek Tiwari > Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Provide observability about how much work remains for a job. In addition, > enable measurement and analysis about how long sub-tasks (`WorkUnit`s) take > while processing some amount of data. > We wish this info to feed into the analysis for Gobblin-on-Temporal (GoT) > auto-scaling and right-sizing, based on Dynamic Scaling - > https://issues.apache.org/jira/browse/GOBBLIN-2170 > e.g. in `GenerateWorkUnits`' result, capture the total number of > `MultiWorkUnit`s and constituent sub-`WorkUnit`s, their total size in bytes, > as well as the size distribution across all WUs. > also, in the `WorkUnitClaimCheck` argument to each `ProcessWorkUnit` > activity, capture the total size of the possibly `MultiWorkUnit` as well as > an abbreviated characterization of the size of any constituent `WorkUnit`s. -- This message was sent by Atlassian Jira (v8.20.10#820010)