Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
yihua merged PR #11035: URL: https://github.com/apache/hudi/pull/11035 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2109996567 ## CI report: * 2efa695bfd6f6a6fc476c8610c7d2d1cce1e965a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23912) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2109793187 ## CI report: * 13176f2a18650a5e189b39c340ed33c004a36d0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23908) * 2efa695bfd6f6a6fc476c8610c7d2d1cce1e965a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23912) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2109776096 ## CI report: * 13176f2a18650a5e189b39c340ed33c004a36d0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23908) * 2efa695bfd6f6a6fc476c8610c7d2d1cce1e965a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2109663327 ## CI report: * 13176f2a18650a5e189b39c340ed33c004a36d0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23908) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2109643741 ## CI report: * 074845c216002fc00c28dcbb7720ffc05bdc7e8f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23891) * 13176f2a18650a5e189b39c340ed33c004a36d0c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
nsivabalan commented on code in PR #11035:
URL: https://github.com/apache/hudi/pull/11035#discussion_r1599621449
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##
@@ -141,8 +141,8 @@ public JavaRDD upsert(JavaRDD>
records, String inst
preWrite(instantTime, WriteOperationType.UPSERT, table.getMetaClient());
HoodieWriteMetadata> result =
table.upsert(context, instantTime, HoodieJavaRDD.of(records));
HoodieWriteMetadata> resultRDD =
result.clone(HoodieJavaRDD.getJavaRDD(result.getWriteStatuses()));
-if (result.getIndexLookupDuration().isPresent()) {
- metrics.updateIndexMetrics(LOOKUP_STR,
result.getIndexLookupDuration().get().toMillis());
+if (result.getSourceReadAndIndexDurationMs().isPresent()) {
+ metrics.updateSourceReadAndIndexMetrics(LOOKUP_STR,
result.getSourceReadAndIndexDurationMs().get());
Review Comment:
Duration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
nsivabalan commented on code in PR #11035:
URL: https://github.com/apache/hudi/pull/11035#discussion_r1599543244
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseWriteHelper.java:
##
@@ -46,22 +47,31 @@ public HoodieWriteMetadata write(String instantTime,
int configuredShuffleParallelism,
BaseCommitActionExecutor
executor,
WriteOperationType operationType) {
+return this.write(instantTime, inputRecords, context, table,
shouldCombine, configuredShuffleParallelism, executor, operationType,
Option.empty());
+ }
+
+ public HoodieWriteMetadata write(String instantTime,
+ I inputRecords,
+ HoodieEngineContext context,
+ HoodieTable table,
+ boolean shouldCombine,
+ int configuredShuffleParallelism,
+ BaseCommitActionExecutor
executor,
+ WriteOperationType operationType,
+ Option
sourceReadAndIndexTimer) {
try {
// De-dupe/merge if needed
I dedupedRecords =
combineOnCondition(shouldCombine, inputRecords,
configuredShuffleParallelism, table);
- Instant lookupBegin = Instant.now();
I taggedRecords = dedupedRecords;
Review Comment:
makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
nsivabalan commented on code in PR #11035:
URL: https://github.com/apache/hudi/pull/11035#discussion_r1599543477
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseCommitActionExecutor.java:
##
@@ -112,6 +113,10 @@ public BaseCommitActionExecutor(HoodieEngineContext
context, HoodieWriteConfig c
public abstract HoodieWriteMetadata execute(I inputRecords);
+ public HoodieWriteMetadata execute(I inputRecords, Option
sourceReadAndIndexTimer) {
+return this.execute(inputRecords);
Review Comment:
I have fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2109223951 ## CI report: * 074845c216002fc00c28dcbb7720ffc05bdc7e8f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23891) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
danny0405 commented on code in PR #11035:
URL: https://github.com/apache/hudi/pull/11035#discussion_r1599312674
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseCommitActionExecutor.java:
##
@@ -112,6 +113,10 @@ public BaseCommitActionExecutor(HoodieEngineContext
context, HoodieWriteConfig c
public abstract HoodieWriteMetadata execute(I inputRecords);
+ public HoodieWriteMetadata execute(I inputRecords, Option
sourceReadAndIndexTimer) {
+return this.execute(inputRecords);
Review Comment:
Not sure why we need a new `#execute` interface, I see that all the impl
executors initialize the timer on the fly while invoking this method, so why
not just initialize the timer in the `#execute`itself.
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/HoodieWriteMetadata.java:
##
@@ -34,6 +34,7 @@ public class HoodieWriteMetadata {
private O writeStatuses;
private Option indexLookupDuration = Option.empty();
Review Comment:
Should we remove this?
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseWriteHelper.java:
##
@@ -46,22 +47,31 @@ public HoodieWriteMetadata write(String instantTime,
int configuredShuffleParallelism,
BaseCommitActionExecutor
executor,
WriteOperationType operationType) {
+return this.write(instantTime, inputRecords, context, table,
shouldCombine, configuredShuffleParallelism, executor, operationType,
Option.empty());
+ }
+
+ public HoodieWriteMetadata write(String instantTime,
+ I inputRecords,
+ HoodieEngineContext context,
+ HoodieTable table,
+ boolean shouldCombine,
+ int configuredShuffleParallelism,
+ BaseCommitActionExecutor
executor,
+ WriteOperationType operationType,
+ Option
sourceReadAndIndexTimer) {
try {
// De-dupe/merge if needed
I dedupedRecords =
combineOnCondition(shouldCombine, inputRecords,
configuredShuffleParallelism, table);
- Instant lookupBegin = Instant.now();
I taggedRecords = dedupedRecords;
Review Comment:
Same question, why not just initialzie the timer here so that we can avoid
to introduce a new method.
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##
@@ -141,8 +141,8 @@ public JavaRDD upsert(JavaRDD>
records, String inst
preWrite(instantTime, WriteOperationType.UPSERT, table.getMetaClient());
HoodieWriteMetadata> result =
table.upsert(context, instantTime, HoodieJavaRDD.of(records));
HoodieWriteMetadata> resultRDD =
result.clone(HoodieJavaRDD.getJavaRDD(result.getWriteStatuses()));
-if (result.getIndexLookupDuration().isPresent()) {
- metrics.updateIndexMetrics(LOOKUP_STR,
result.getIndexLookupDuration().get().toMillis());
+if (result.getSourceReadAndIndexDurationMs().isPresent()) {
+ metrics.updateSourceReadAndIndexMetrics(LOOKUP_STR,
result.getSourceReadAndIndexDurationMs().get());
Review Comment:
Should we still use `LOOKUP_STR` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2109136722 ## CI report: * e0d1d604a6331759903f4e825499f89afaac1d00 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23880) * 074845c216002fc00c28dcbb7720ffc05bdc7e8f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23891) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
danny0405 commented on code in PR #11035:
URL: https://github.com/apache/hudi/pull/11035#discussion_r1599280721
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/HoodieMetrics.java:
##
@@ -207,6 +210,13 @@ public Timer.Context getIndexCtx() {
return indexTimer == null ? null : indexTimer.time();
}
+ public Timer.Context getPreWriteTimerCtx() {
+if (config.isMetricsOn() && preWriteTimer == null) {
+ preWriteTimer = createTimer(preWriteTimerName);
+}
Review Comment:
+1 for `source_read_and_index`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2109130452 ## CI report: * e0d1d604a6331759903f4e825499f89afaac1d00 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23880) * 074845c216002fc00c28dcbb7720ffc05bdc7e8f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2108878120 ## CI report: * e0d1d604a6331759903f4e825499f89afaac1d00 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23880) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2108774485 ## CI report: * 1774e4cfb0f03bcc056a18745e19ff6508c2c07a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23879) * e0d1d604a6331759903f4e825499f89afaac1d00 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23880) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2108699292 ## CI report: * 244e2a201eb3b482089a30812f5ff53065ac8918 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23289) * 1774e4cfb0f03bcc056a18745e19ff6508c2c07a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23879) * e0d1d604a6331759903f4e825499f89afaac1d00 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2108687973 ## CI report: * 244e2a201eb3b482089a30812f5ff53065ac8918 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23289) * 1774e4cfb0f03bcc056a18745e19ff6508c2c07a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23879) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2108675675 ## CI report: * 244e2a201eb3b482089a30812f5ff53065ac8918 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23289) * 1774e4cfb0f03bcc056a18745e19ff6508c2c07a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
nsivabalan commented on code in PR #11035:
URL: https://github.com/apache/hudi/pull/11035#discussion_r1599000442
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/HoodieMetrics.java:
##
@@ -207,6 +210,13 @@ public Timer.Context getIndexCtx() {
return indexTimer == null ? null : indexTimer.time();
}
+ public Timer.Context getPreWriteTimerCtx() {
+if (config.isMetricsOn() && preWriteTimer == null) {
+ preWriteTimer = createTimer(preWriteTimerName);
+}
Review Comment:
sure. sg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
yihua commented on code in PR #11035:
URL: https://github.com/apache/hudi/pull/11035#discussion_r1598982601
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/HoodieMetrics.java:
##
@@ -207,6 +210,13 @@ public Timer.Context getIndexCtx() {
return indexTimer == null ? null : indexTimer.time();
}
+ public Timer.Context getPreWriteTimerCtx() {
+if (config.isMetricsOn() && preWriteTimer == null) {
+ preWriteTimer = createTimer(preWriteTimerName);
+}
Review Comment:
@danny0405 this metric also contains the latency of reading the data from
source. So should we rename it to `source_read_and_index`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
yihua commented on code in PR #11035:
URL: https://github.com/apache/hudi/pull/11035#discussion_r1598982601
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/HoodieMetrics.java:
##
@@ -207,6 +210,13 @@ public Timer.Context getIndexCtx() {
return indexTimer == null ? null : indexTimer.time();
}
+ public Timer.Context getPreWriteTimerCtx() {
+if (config.isMetricsOn() && preWriteTimer == null) {
+ preWriteTimer = createTimer(preWriteTimerName);
+}
Review Comment:
@danny0405 this metric also contains the latency of reading the data from
source. So should we rename it to `read_source_and_index`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
yihua commented on code in PR #11035:
URL: https://github.com/apache/hudi/pull/11035#discussion_r1598982601
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/HoodieMetrics.java:
##
@@ -207,6 +210,13 @@ public Timer.Context getIndexCtx() {
return indexTimer == null ? null : indexTimer.time();
}
+ public Timer.Context getPreWriteTimerCtx() {
+if (config.isMetricsOn() && preWriteTimer == null) {
+ preWriteTimer = createTimer(preWriteTimerName);
+}
Review Comment:
@danny0405 this metric also contains the latency of reading the data from
source. So should we rename it to `reading_source_and_indexing`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
danny0405 commented on code in PR #11035:
URL: https://github.com/apache/hudi/pull/11035#discussion_r1590626909
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/HoodieMetrics.java:
##
@@ -207,6 +210,13 @@ public Timer.Context getIndexCtx() {
return indexTimer == null ? null : indexTimer.time();
}
+ public Timer.Context getPreWriteTimerCtx() {
+if (config.isMetricsOn() && preWriteTimer == null) {
+ preWriteTimer = createTimer(preWriteTimerName);
+}
Review Comment:
Looks like a indexing time for the records, how about name it as just
"indexing".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
nsivabalan commented on code in PR #11035:
URL: https://github.com/apache/hudi/pull/11035#discussion_r1590607676
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/HoodieMetrics.java:
##
@@ -207,6 +210,13 @@ public Timer.Context getIndexCtx() {
return indexTimer == null ? null : indexTimer.time();
}
+ public Timer.Context getPreWriteTimerCtx() {
+if (config.isMetricsOn() && preWriteTimer == null) {
+ preWriteTimer = createTimer(preWriteTimerName);
+}
Review Comment:
sure. do you have any good suggestion for this metric name.
As you might be aware, it spans from reading from source up until completion
of tag location.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
danny0405 commented on code in PR #11035:
URL: https://github.com/apache/hudi/pull/11035#discussion_r1568172191
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/HoodieMetrics.java:
##
@@ -207,6 +210,13 @@ public Timer.Context getIndexCtx() {
return indexTimer == null ? null : indexTimer.time();
}
+ public Timer.Context getPreWriteTimerCtx() {
+if (config.isMetricsOn() && preWriteTimer == null) {
+ preWriteTimer = createTimer(preWriteTimerName);
+}
Review Comment:
`preWrite` is not a meaningful terminology, don't think we should expose an
metrics for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2059983018 ## CI report: * 244e2a201eb3b482089a30812f5ff53065ac8918 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23289) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2059929841 ## CI report: * 244e2a201eb3b482089a30812f5ff53065ac8918 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23289) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7624] Fixing index tagging duration [hudi]
hudi-bot commented on PR #11035: URL: https://github.com/apache/hudi/pull/11035#issuecomment-2059920162 ## CI report: * 244e2a201eb3b482089a30812f5ff53065ac8918 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
