[
https://issues.apache.org/jira/browse/HUDI-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinish Reddy updated HUDI-3788:
-------------------------------
Description:
{{Improvements in observability/metrics for Hudi OSS}}
{{1.HoodieCleanMetadata}} contains only the number of files deleted and the
name of the files but doesn't contain the total bytes that have been cleaned
up. This is an important metric to keep track of and total size in bytes
metrics if computed will always be more and will keep missing the bytes removed
by the cleaner.
2. In HoodieWriteStat, we don't have a metric to know how much data is being
added/removed as part of each file slice being written. If we want to know how
many bytes are being added/remove in the table as part of each commit we need
this metric as part of HoodieWriteStat.
3. latency fields - totalUpsertTime, totalCreateTime, totalScanTime are missing
in archived HoodieCommitMetadata. This makes bootstrapping the metrics for a
table having archived commits difficult and not possible unless we have the
active commit me
was:{{HoodieCleanMetadata}} contains only the number of files deleted and the
name of the files but doesn't contain the total bytes that have been cleaned
up. This is an important metric to keep track of and total size in bytes
metrics if computed will always be more and will keep missing the bytes removed
by the cleaner.
> Persist totalBytesWritten as part of HoodieCleanMetadata
> --------------------------------------------------------
>
> Key: HUDI-3788
> URL: https://issues.apache.org/jira/browse/HUDI-3788
> Project: Apache Hudi
> Issue Type: Task
> Reporter: Vinish Reddy
> Priority: Major
>
> {{Improvements in observability/metrics for Hudi OSS}}
> {{1.HoodieCleanMetadata}} contains only the number of files deleted and the
> name of the files but doesn't contain the total bytes that have been cleaned
> up. This is an important metric to keep track of and total size in bytes
> metrics if computed will always be more and will keep missing the bytes
> removed by the cleaner.
> 2. In HoodieWriteStat, we don't have a metric to know how much data is being
> added/removed as part of each file slice being written. If we want to know
> how many bytes are being added/remove in the table as part of each commit we
> need this metric as part of HoodieWriteStat.
> 3. latency fields - totalUpsertTime, totalCreateTime, totalScanTime are
> missing in archived HoodieCommitMetadata. This makes bootstrapping the
> metrics for a table having archived commits difficult and not possible unless
> we have the active commit me
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
