Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
danny0405 merged PR #13403: URL: https://github.com/apache/hudi/pull/13403 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2952790516 ## CI report: * 4982230ffccf98a47ff4336ea190ed3400d85ebd UNKNOWN * 79dd2854bd50b6c27d12cdf0cc3e4d90440d5bde UNKNOWN * 0512ef6e841243d2363f819f1be8b433bae191fc Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6070) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2952705890 ## CI report: * fffdfbb4812e87ca4884636b1286ed9ccd52cf04 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6069) * 4982230ffccf98a47ff4336ea190ed3400d85ebd UNKNOWN * 79dd2854bd50b6c27d12cdf0cc3e4d90440d5bde UNKNOWN * 0512ef6e841243d2363f819f1be8b433bae191fc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6070) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2952678745 ## CI report: * fffdfbb4812e87ca4884636b1286ed9ccd52cf04 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6069) * 4982230ffccf98a47ff4336ea190ed3400d85ebd UNKNOWN * 79dd2854bd50b6c27d12cdf0cc3e4d90440d5bde UNKNOWN * 0512ef6e841243d2363f819f1be8b433bae191fc UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2952647531 ## CI report: * fffdfbb4812e87ca4884636b1286ed9ccd52cf04 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6069) * 4982230ffccf98a47ff4336ea190ed3400d85ebd UNKNOWN * 79dd2854bd50b6c27d12cdf0cc3e4d90440d5bde UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2952642393 ## CI report: * fffdfbb4812e87ca4884636b1286ed9ccd52cf04 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6069) * 4982230ffccf98a47ff4336ea190ed3400d85ebd UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2952640278 ## CI report: * 72236f3c3f30c3b83ca230a1758579264576826a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6068) * fffdfbb4812e87ca4884636b1286ed9ccd52cf04 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6069) * 4982230ffccf98a47ff4336ea190ed3400d85ebd UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2952611723 ## CI report: * 72236f3c3f30c3b83ca230a1758579264576826a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6068) * fffdfbb4812e87ca4884636b1286ed9ccd52cf04 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6069) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2952606999 ## CI report: * 72236f3c3f30c3b83ca230a1758579264576826a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6068) * fffdfbb4812e87ca4884636b1286ed9ccd52cf04 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2951906791 ## CI report: * 72236f3c3f30c3b83ca230a1758579264576826a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6068) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2951674358 ## CI report: * bbd78fe2010b0d2bf4a15b0112a6d74e3ac2a68d Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6067) * 72236f3c3f30c3b83ca230a1758579264576826a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6068) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2951571679 ## CI report: * bbd78fe2010b0d2bf4a15b0112a6d74e3ac2a68d Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6067) * 72236f3c3f30c3b83ca230a1758579264576826a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2951568605 ## CI report: * f9eeb0f78bc8bcd7b2b61784a8332a50527d96b9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6061) * bbd78fe2010b0d2bf4a15b0112a6d74e3ac2a68d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6067) * 72236f3c3f30c3b83ca230a1758579264576826a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2951562933 ## CI report: * f9eeb0f78bc8bcd7b2b61784a8332a50527d96b9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6061) * bbd78fe2010b0d2bf4a15b0112a6d74e3ac2a68d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
danny0405 commented on code in PR #13403:
URL: https://github.com/apache/hudi/pull/13403#discussion_r2133203943
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java:
##
@@ -549,6 +550,9 @@ private void doCommit(long checkpointId, String instant,
List write
if (!hasErrors || this.conf.get(FlinkOptions.IGNORE_FAILED)) {
HashMap checkpointCommitMetadata = new HashMap<>();
+ if (this.conf.get(FlinkOptions.EXTRA_METADATA_ENABLED)) {
Review Comment:
Can we add an utility method in `StreamerUtils` to achieve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
danny0405 commented on code in PR #13403:
URL: https://github.com/apache/hudi/pull/13403#discussion_r2133203778
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##
@@ -163,6 +163,13 @@ private FlinkOptions() {
.withDescription("The default partition name in case the dynamic
partition"
+ " column value is null/empty string");
+ @AdvancedConfig
+ public static final ConfigOption EXTRA_METADATA_ENABLED =
ConfigOptions
+ .key("extra.metadata.enabled")
Review Comment:
write.extra.metadata.enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2951326158 ## CI report: * f9eeb0f78bc8bcd7b2b61784a8332a50527d96b9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6061) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2951164870 ## CI report: * 6410882eefd4e454475a9ad9b64ff2458cc3380b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6057) * f9eeb0f78bc8bcd7b2b61784a8332a50527d96b9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6061) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2950920365 ## CI report: * 6410882eefd4e454475a9ad9b64ff2458cc3380b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2950930071 ## CI report: * 6410882eefd4e454475a9ad9b64ff2458cc3380b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6057) * f9eeb0f78bc8bcd7b2b61784a8332a50527d96b9 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2950925159 ## CI report: * 6410882eefd4e454475a9ad9b64ff2458cc3380b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6057) * f9eeb0f78bc8bcd7b2b61784a8332a50527d96b9 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-9491] add checkpoint id into hudi commit metadata [hudi]
hudi-bot commented on PR #13403: URL: https://github.com/apache/hudi/pull/13403#issuecomment-2950922862 ## CI report: * 6410882eefd4e454475a9ad9b64ff2458cc3380b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6057) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
