pratyakshsharma commented on a change in pull request #1647:
URL: https://github.com/apache/incubator-hudi/pull/1647#discussion_r429158430
##########
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java
##########
@@ -416,10 +425,12 @@ public DeltaSync getDeltaSync() {
jssc.setLocalProperty("spark.scheduler.pool",
SchedulerConfGenerator.DELTASYNC_POOL_NAME);
}
try {
+ int iteration = 1;
while (!isShutdownRequested()) {
try {
long start = System.currentTimeMillis();
- Option<String> scheduledCompactionInstant = deltaSync.syncOnce();
+ HoodieMetrics.setTableName(cfg.metricsTableName + "_" +
iteration);
Review comment:
IllegalArgumentException is happening because the metrics name are
generated in same way using tableName in each run. So we need some way of
differentiating metrics names for every run and the easiest way to do that is
altering the table name like "<tableName>_<iteration>". We need to do this
change at 2 places, for HoodieDeltaStreamerMetrics and for HoodieMetrics.
The table name getting passed with syncOnce() method takes care of
HoodieDeltaStreamerMetrics only. For all the other metrics, we need to reset
table name in HoodieMetrics class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
