AlexKhan8 opened a new issue, #8427: URL: https://github.com/apache/hudi/issues/8427
**_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at [email protected]. - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly. **Describe the problem you faced** Number of documented [metrics](https://hudi.apache.org/docs/metrics/#list-of-metrics) aren't showing up in cloudwatch when the documented configuration is added. **To Reproduce** Steps to reproduce the behavior: 1. Added config to produce metrics 2. Checked cloudwatch for metric (i.e commitFreshnessInMS commitLatencyInMs) but metric is not produced. Here are the metrics we do see populated.    **Expected behavior** Added config to produce the metrics documented. **Environment Description** * Hudi version : 0.12.1 * EMR version : 6.8.9 * Storage (HDFS/S3/GCS..) : S3 * Running on Docker? (yes/no) : no **Additional context** Add any other context about the problem here. **Configuration** We've added the hoodie.payload.event.time.field config and relevant metrics configuration as shown below ``` "--hoodie-conf", "hoodie.database.name={{ database }}_raw", "--hoodie-conf", "hoodie.table.name={{ table }}", "--hoodie-conf", "hoodie.datasource.write.recordkey.field={{ primary_key }}", "--hoodie-conf", "hoodie.datasource.write.precombine.field={{ timestamp_field }}", "--hoodie-conf", "hoodie.payload.event.time.field={{ timestamp_field }}", "--hoodie-conf", "hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.TimestampBasedKeyGenerator", "--hoodie-conf", "hoodie.datasource.write.partitionpath.field={{ timestamp_field }}", "--hoodie-conf", "hoodie.deltastreamer.keygen.timebased.timestamp.type=EPOCHMILLISECONDS", "--hoodie-conf", "hoodie.deltastreamer.keygen.timebased.output.dateformat=yyyy/MM/dd", "--hoodie-conf", "hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled=true", "--hoodie-conf", "hoodie.deltastreamer.source.kafka.value.deserializer.class=org.apache.hudi.utilities.deser.KafkaAvroSchemaDeserializer", # AWS Glue Data Catalog config "--hoodie-conf", "hoodie.datasource.hive_sync.enable=true", "--hoodie-conf", "hoodie.datasource.hive_sync.database={{ database }}_raw", "--hoodie-conf", "hoodie.datasource.hive_sync.table={{ table }}", "--hoodie-conf", "hoodie.datasource.hive_sync.partition_fields=_event_date", # Hudi Metrics "--hoodie-conf", "hoodie.metrics.on=true", "--hoodie-conf", "hoodie.metrics.reporter.type=CLOUDWATCH", ``` When checking the commit log we see that the minAndMaxEventTime is not being populated: ``` "totalRecordsDeleted" : 0, "totalLogRecordsCompacted" : 0, "totalLogFilesCompacted" : 0, "totalCompactedRecordsUpdated" : 0, "totalLogFilesSize" : 0, "totalScanTime" : 0, "totalCreateTime" : 0, "totalUpsertTime" : 1768, "minAndMaxEventTime" : { "Optional.empty" : { "val" : null, "present" : false } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
