[jira] [Resolved] (HUDI-2204) Add marker files for flink writer
[ https://issues.apache.org/jira/browse/HUDI-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-2204. -- Resolution: Fixed Fixed via master branch: 2370a9facbe4418f994f29c426e9b2a255e3abb0 > Add marker files for flink writer > - > > Key: HUDI-2204 > URL: https://issues.apache.org/jira/browse/HUDI-2204 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2204) Add marker files for flink writer
[ https://issues.apache.org/jira/browse/HUDI-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385266#comment-17385266 ] ASF GitHub Bot commented on HUDI-2204: -- danny0405 merged pull request #3316: URL: https://github.com/apache/hudi/pull/3316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add marker files for flink writer > - > > Key: HUDI-2204 > URL: https://issues.apache.org/jira/browse/HUDI-2204 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[hudi] branch master updated (5a94b6b -> 2370a9f)
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 5a94b6b [HUDI-2192] Clean up Multiple versions of scala libraries detected Warning (#3292) add 2370a9f [HUDI-2204] Add marker files for flink writer (#3316) No new revisions were added by this update. Summary of changes: .../hudi/client/common/HoodieFlinkEngineContext.java | 2 +- .../java/org/apache/hudi/io/FlinkCreateHandle.java| 7 +++ .../apache/hudi/io/FlinkMergeAndReplaceHandle.java| 4 +++- .../java/org/apache/hudi/io/FlinkMergeHandle.java | 6 +++--- .../hudi/table/HoodieFlinkCopyOnWriteTable.java | 15 --- .../table/action/clean/FlinkCleanActionExecutor.java | 19 --- 6 files changed, 22 insertions(+), 31 deletions(-)
[GitHub] [hudi] danny0405 merged pull request #3316: [HUDI-2204] Add marker files for flink writer
danny0405 merged pull request #3316: URL: https://github.com/apache/hudi/pull/3316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan
[ https://issues.apache.org/jira/browse/HUDI-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385262#comment-17385262 ] ASF GitHub Bot commented on HUDI-2194: -- zhangyue19921010 commented on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-884662878 Hi @satishkotha Thanks for your review. > @zhangyue19921010 LGTM. Is it possible to add a test? Sure thing, added. PTAL :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Skip the latest N partitions when creating ClusteringPlan > - > > Key: HUDI-2194 > URL: https://issues.apache.org/jira/browse/HUDI-2194 > Project: Apache Hudi > Issue Type: Task >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > > As we known, SparkRecentDaysClusteringPlanStrategy is the default clustering > strategy to create ClusteringPlan. And it is useful when Hudi table is > partitioned by time. > > For now, users can set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` to control > the number of partitions to list from the latest partition to create > ClusteringPlan. > For example, we have 6 partitions based on date, and users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > > |<- choose to cluster >| > Sometimes users also what to skip x partitions from latest when make > clustering plan because latest partitions contains lots of update data or > some reasons else. > > This patch will add a new config named ` > hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions > ` to set the > number of partitions to skip from latest when choosing partitions to create > ClusteringPlan > > for example users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 and > ` > hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions > ` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > |<- choose ->| > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] zhangyue19921010 commented on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan
zhangyue19921010 commented on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-884662878 Hi @satishkotha Thanks for your review. > @zhangyue19921010 LGTM. Is it possible to add a test? Sure thing, added. PTAL :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan
[ https://issues.apache.org/jira/browse/HUDI-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385261#comment-17385261 ] ASF GitHub Bot commented on HUDI-2194: -- codecov-commenter edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882457453 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Skip the latest N partitions when creating ClusteringPlan > - > > Key: HUDI-2194 > URL: https://issues.apache.org/jira/browse/HUDI-2194 > Project: Apache Hudi > Issue Type: Task >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > > As we known, SparkRecentDaysClusteringPlanStrategy is the default clustering > strategy to create ClusteringPlan. And it is useful when Hudi table is > partitioned by time. > > For now, users can set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` to control > the number of partitions to list from the latest partition to create > ClusteringPlan. > For example, we have 6 partitions based on date, and users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > > |<- choose to cluster >| > Sometimes users also what to skip x partitions from latest when make > clustering plan because latest partitions contains lots of update data or > some reasons else. > > This patch will add a new config named ` > hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions > ` to set the > number of partitions to skip from latest when choosing partitions to create > ClusteringPlan > > for example users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 and > ` > hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions > ` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > |<- choose ->| > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan
codecov-commenter edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882457453 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan
[ https://issues.apache.org/jira/browse/HUDI-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385260#comment-17385260 ] ASF GitHub Bot commented on HUDI-2194: -- hudi-bot edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882429208 ## CI report: * 4b85d50e5c18695641d1c91edccf6a638260761a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1092) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Skip the latest N partitions when creating ClusteringPlan > - > > Key: HUDI-2194 > URL: https://issues.apache.org/jira/browse/HUDI-2194 > Project: Apache Hudi > Issue Type: Task >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > > As we known, SparkRecentDaysClusteringPlanStrategy is the default clustering > strategy to create ClusteringPlan. And it is useful when Hudi table is > partitioned by time. > > For now, users can set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` to control > the number of partitions to list from the latest partition to create > ClusteringPlan. > For example, we have 6 partitions based on date, and users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > > |<- choose to cluster >| > Sometimes users also what to skip x partitions from latest when make > clustering plan because latest partitions contains lots of update data or > some reasons else. > > This patch will add a new config named ` > hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions > ` to set the > number of partitions to skip from latest when choosing partitions to create > ClusteringPlan > > for example users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 and > ` > hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions > ` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > |<- choose ->| > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan
hudi-bot edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882429208 ## CI report: * 4b85d50e5c18695641d1c91edccf6a638260761a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1092) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] vinothchandar closed issue #3236: why use avro not parquet in MOR
vinothchandar closed issue #3236: URL: https://github.com/apache/hudi/issues/3236 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan
omments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...pache/hudi/client/utils/ConcatenatingIterator.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC91dGlscy9Db25jYXRlbmF0aW5nSXRlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | ... and [650 more](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=continue_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=footer_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). Last update [572a214...4b85d50](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=lastupdated_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Skip the latest N partitions when creating ClusteringPlan > - > > Key: HUDI-2194 > URL: https://issues.apache.org/jira/browse/HUDI-2194 > Project: Apache Hudi > Issue Type: Task >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > > As we known, SparkRecentDaysClusteringPlanStrategy is the default clustering > strategy to create ClusteringPlan. And it is useful when Hudi table is > partitioned by time. > > For now, users can set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` to control > the number of partitions to list from the latest partition to create > ClusteringPlan. > For example, we have 6 partitions based on date, and users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > > |<- choose to cluster >| > Sometimes users also what
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan
codecov-commenter edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882457453 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3300](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (4b85d50) into [master](https://codecov.io/gh/apache/hudi/commit/572a2144124b90b2dbef8183ee19a95b21de2716?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (572a214) will **decrease** coverage by `20.18%`. > The diff coverage is `20.00%`. [](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3300 +/- ## = - Coverage 47.83% 27.65% -20.19% + Complexity 5565 1324 -4241 = Files 936 390 -546 Lines 4166315597-26066 Branches 4197 1381 -2816 = - Hits 19929 4313-15616 + Misses1996010959 -9001 + Partials 1774 325 -1449 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.19% <20.00%> (-13.33%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.84% <ø> (+0.07%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...org/apache/hudi/config/HoodieClusteringConfig.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVDbHVzdGVyaW5nQ29uZmlnLmphdmE=) | `0.00% <0.00%> (-71.57%)` | :arrow_down: | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.52%)` | :arrow_down: | | [...trategy/SparkRecentDaysClusteringPlanStrategy.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2NsdXN0ZXJpbmcvcGxhbi9zdHJhdGVneS9TcGFya1JlY2VudERheXNDbHVzdGVyaW5nUGxhblN0cmF0ZWd5LmphdmE=) | `76.74% <100.00%> (+1.13%)` | :arrow_up: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3325: [WIP] Fixing payload instantiation to include preCombine field in LogRecordScanner
hudi-bot edited a comment on pull request #3325: URL: https://github.com/apache/hudi/pull/3325#issuecomment-884643521 ## CI report: * 7ab74d46bc22b7e93e0ac00a2200b62c4bd120e6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1091) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan
mE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/metrics/MetricsReporterType.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/client/bootstrap/BootstrapMode.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9ib290c3RyYXAvQm9vdHN0cmFwTW9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...he/hudi/hive/HiveStylePartitionValueExtractor.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN0eWxlUGFydGl0aW9uVmFsdWVFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...pache/hudi/client/utils/ConcatenatingIterator.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC91dGlscy9Db25jYXRlbmF0aW5nSXRlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...che/hudi/config/HoodieMetricsPrometheusConfig.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVNZXRyaWNzUHJvbWV0aGV1c0NvbmZpZy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | ... and [725 more](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=continue_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=footer_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). Last update [572a214...4b85d50](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=lastupdated_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Skip the latest N partitions when creating ClusteringPlan > - > > Key: HUDI-2194 > URL: https://issues.apache.org/jira/browse/HUDI-2194 > Project: Apache Hudi > Issue Type: Task >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > > As we known, SparkRecentDaysClusteringPlanStrategy is the default clustering > strategy to create ClusteringPlan. And it is useful when Hudi table is > partitioned by time. > > For now, users can set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` to control > the number of partitions to list from the latest partition to create > ClusteringPlan. > For example, we have 6 partitions based on date, and users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > > |<- choose to cluster >| > Sometimes users also what to skip x pa
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan
codecov-commenter edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882457453 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3300](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (4b85d50) into [master](https://codecov.io/gh/apache/hudi/commit/572a2144124b90b2dbef8183ee19a95b21de2716?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (572a214) will **decrease** coverage by `31.76%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3300 +/- ## = - Coverage 47.83% 16.07% -31.77% + Complexity 5565 504 -5061 = Files 936 284 -652 Lines 4166311889-29774 Branches 4197 986 -3211 = - Hits 19929 1911-18018 + Misses19960 9813-10147 + Partials 1774 165 -1609 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.53%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.84% <ø> (+0.07%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...org/apache/hudi/config/HoodieClusteringConfig.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVDbHVzdGVyaW5nQ29uZmlnLmphdmE=) | `0.00% <0.00%> (-71.57%)` | :arrow_down: | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.52%)` | :arrow_down: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/hive/NonPartitionedExtractor.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvTm9uUGFydGl0aW9uZWRFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../java/org/apache/hudi/metrics/MetricsReporter.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvTWV0cmljc1JlcG9ydGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385251#comment-17385251 ] ASF GitHub Bot commented on HUDI-1771: -- hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * a46c6f21415c65e005ba9ac267cd2bda4528a39b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1089) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Propagate CDC format for hoodie > --- > > Key: HUDI-1771 > URL: https://issues.apache.org/jira/browse/HUDI-1771 > Project: Apache Hudi > Issue Type: New Feature > Components: Flink Integration >Reporter: Danny Chen >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available, sev:normal > Fix For: 0.9.0 > > > Like what we discussed in the dev mailing list: > https://lists.apache.org/thread.html/r31b2d1404e4e043a5f875b78105ba6f9a801e78f265ad91242ad5eb2%40%3Cdev.hudi.apache.org%3E > Keep the change flags make new use cases possible: using HUDI as the unified > storage format for DWD and DWS layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie
hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * a46c6f21415c65e005ba9ac267cd2bda4528a39b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1089) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3313: [SUPPORT] CoW: Hudi Upsert not working when there is a timestamp field in the composite key
nsivabalan commented on issue #3313: URL: https://github.com/apache/hudi/issues/3313#issuecomment-884649681 Would you mind giving me steps to reproduce w/ some sample data. guess handful of records should be good enough to reproduce. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3313: [SUPPORT] CoW: Hudi Upsert not working when there is a timestamp field in the composite key
nsivabalan commented on issue #3313: URL: https://github.com/apache/hudi/issues/3313#issuecomment-884649224 AFAIK, CustomKeyGenerator does pick field values as is. don't think it does any timestamp related conversion. Can you confirm that during bulk_insert the claim_subm_dt field value was "2020-11-21 00:00:00.0", and while you upserted it was "16059168". I mean in the incoming df. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan
w_down: | | [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | ... and [772 more](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=continue_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=footer_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). Last update [572a214...4b85d50](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=lastupdated_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Skip the latest N partitions when creating ClusteringPlan > - > > Key: HUDI-2194 > URL: https://issues.apache.org/jira/browse/HUDI-2194 > Project: Apache Hudi > Issue Type: Task >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > > As we known, SparkRecentDaysClusteringPlanStrategy is the default clustering > strategy to create ClusteringPlan. And it is useful when Hudi table is > partitioned by time. > > For now, users can set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` to control > the number of partitions to list from the latest partition to create > ClusteringPlan. > For example, we have 6 partitions based on date, and users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > > |<- choose to cluster >| > Sometimes users also what to skip x partitions from latest when make > clustering plan because latest partitions contains lots of update data or > some reasons else. > > This patch will add a new config named ` > hoodie.cluste
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan
codecov-commenter edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882457453 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3300](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (4b85d50) into [master](https://codecov.io/gh/apache/hudi/commit/572a2144124b90b2dbef8183ee19a95b21de2716?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (572a214) will **decrease** coverage by `45.01%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3300 +/- ## - Coverage 47.83% 2.81% -45.02% + Complexity 5565 85 -5480 Files 936 284 -652 Lines 41663 11889-29774 Branches 4197 986 -3211 - Hits 19929 335-19594 + Misses19960 11528 -8432 + Partials 1774 26 -1748 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.53%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.97% <ø> (-50.80%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3300?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...org/apache/hudi/config/HoodieClusteringConfig.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVDbHVzdGVyaW5nQ29uZmlnLmphdmE=) | `0.00% <0.00%> (-71.57%)` | :arrow_down: | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.52%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/3300/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | |
[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan
[ https://issues.apache.org/jira/browse/HUDI-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385247#comment-17385247 ] ASF GitHub Bot commented on HUDI-2194: -- hudi-bot edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882429208 ## CI report: * 350f4266dd6951cb64ba492553e1099978f23fcf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1016) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1058) * 4b85d50e5c18695641d1c91edccf6a638260761a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1092) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Skip the latest N partitions when creating ClusteringPlan > - > > Key: HUDI-2194 > URL: https://issues.apache.org/jira/browse/HUDI-2194 > Project: Apache Hudi > Issue Type: Task >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > > As we known, SparkRecentDaysClusteringPlanStrategy is the default clustering > strategy to create ClusteringPlan. And it is useful when Hudi table is > partitioned by time. > > For now, users can set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` to control > the number of partitions to list from the latest partition to create > ClusteringPlan. > For example, we have 6 partitions based on date, and users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > > |<- choose to cluster >| > Sometimes users also what to skip x partitions from latest when make > clustering plan because latest partitions contains lots of update data or > some reasons else. > > This patch will add a new config named ` > hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions > ` to set the > number of partitions to skip from latest when choosing partitions to create > ClusteringPlan > > for example users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 and > ` > hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions > ` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > |<- choose ->| > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan
hudi-bot edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882429208 ## CI report: * 350f4266dd6951cb64ba492553e1099978f23fcf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1016) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1058) * 4b85d50e5c18695641d1c91edccf6a638260761a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1092) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2204) Add marker files for flink writer
[ https://issues.apache.org/jira/browse/HUDI-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385244#comment-17385244 ] ASF GitHub Bot commented on HUDI-2204: -- yuzhaojing commented on pull request #3316: URL: https://github.com/apache/hudi/pull/3316#issuecomment-884645405 +1 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add marker files for flink writer > - > > Key: HUDI-2204 > URL: https://issues.apache.org/jira/browse/HUDI-2204 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] yuzhaojing commented on pull request #3316: [HUDI-2204] Add marker files for flink writer
yuzhaojing commented on pull request #3316: URL: https://github.com/apache/hudi/pull/3316#issuecomment-884645405 +1 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig
codecov-commenter edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883347621 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3307](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (5ad4271) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `2.08%`. > The diff coverage is `97.14%`. [](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3307 +/- ## - Coverage 47.74% 45.65% -2.09% - Complexity 5591 5596 +5 Files 938 999 +61 Lines 4182343790+1967 Branches 4213 4403 +190 + Hits 1996819992 +24 - Misses2007022015+1945 + Partials 1785 1783 -2 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `34.55% <ø> (ø)` | | | hudicommon | `48.65% <ø> (+0.01%)` | :arrow_up: | | hudiflink | `59.62% <100.00%> (+0.18%)` | :arrow_up: | | hudihadoopmr | `52.40% <95.65%> (+0.37%)` | :arrow_up: | | hudiintegtest | `0.00% <ø> (?)` | | | hudisparkdatasource | `67.12% <100.00%> (+0.01%)` | :arrow_up: | | hudisync | `55.97% <ø> (ø)` | | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `59.87% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...pache/hudi/hadoop/config/HoodieRealtimeConfig.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL2NvbmZpZy9Ib29kaWVSZWFsdGltZUNvbmZpZy5qYXZh) | `88.88% <88.88%> (+88.88%)` | :arrow_up: | | [...java/org/apache/hudi/table/format/FormatUtils.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9mb3JtYXQvRm9ybWF0VXRpbHMuamF2YQ==) | `89.65% <100.00%> (-3.68%)` | :arrow_down: | | [...hadoop/realtime/RealtimeCompactedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lQ29tcGFjdGVkUmVjb3JkUmVhZGVyLmphdmE=) | `77.77% <100.00%> (+0.96%)` | :arrow_up: | | [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `97.67% <100.00%> (+0.11%)` | :arrow_up: | | [.../hadoop/utils/HoodieRealtimeRecordReaderUtils.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZVJlYWx0aW1lUmVjb3JkUmVhZGVyVXRpbHMuamF2YQ==) | `72.03% <100.00%> (+0.23%)` | :arrow_up: | |
[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan
[ https://issues.apache.org/jira/browse/HUDI-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385242#comment-17385242 ] ASF GitHub Bot commented on HUDI-2194: -- hudi-bot edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882429208 ## CI report: * 350f4266dd6951cb64ba492553e1099978f23fcf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1016) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1058) * 4b85d50e5c18695641d1c91edccf6a638260761a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Skip the latest N partitions when creating ClusteringPlan > - > > Key: HUDI-2194 > URL: https://issues.apache.org/jira/browse/HUDI-2194 > Project: Apache Hudi > Issue Type: Task >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > > As we known, SparkRecentDaysClusteringPlanStrategy is the default clustering > strategy to create ClusteringPlan. And it is useful when Hudi table is > partitioned by time. > > For now, users can set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` to control > the number of partitions to list from the latest partition to create > ClusteringPlan. > For example, we have 6 partitions based on date, and users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > > |<- choose to cluster >| > Sometimes users also what to skip x partitions from latest when make > clustering plan because latest partitions contains lots of update data or > some reasons else. > > This patch will add a new config named ` > hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions > ` to set the > number of partitions to skip from latest when choosing partitions to create > ClusteringPlan > > for example users set > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 and > ` > hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions > ` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| > |<- choose ->| > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan
hudi-bot edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882429208 ## CI report: * 350f4266dd6951cb64ba492553e1099978f23fcf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1016) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1058) * 4b85d50e5c18695641d1c91edccf6a638260761a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig
codecov-commenter edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883347621 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3307](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (5ad4271) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `2.09%`. > The diff coverage is `97.14%`. [](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3307 +/- ## - Coverage 47.74% 45.64% -2.10% - Complexity 5591 5596 +5 Files 938 999 +61 Lines 4182343790+1967 Branches 4213 4403 +190 + Hits 1996819988 +20 - Misses2007022019+1949 + Partials 1785 1783 -2 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `34.55% <ø> (ø)` | | | hudicommon | `48.65% <ø> (+0.01%)` | :arrow_up: | | hudiflink | `59.62% <100.00%> (+0.18%)` | :arrow_up: | | hudihadoopmr | `52.40% <95.65%> (+0.37%)` | :arrow_up: | | hudiintegtest | `0.00% <ø> (?)` | | | hudisparkdatasource | `67.00% <100.00%> (-0.10%)` | :arrow_down: | | hudisync | `55.97% <ø> (ø)` | | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `59.87% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...pache/hudi/hadoop/config/HoodieRealtimeConfig.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL2NvbmZpZy9Ib29kaWVSZWFsdGltZUNvbmZpZy5qYXZh) | `88.88% <88.88%> (+88.88%)` | :arrow_up: | | [...java/org/apache/hudi/table/format/FormatUtils.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9mb3JtYXQvRm9ybWF0VXRpbHMuamF2YQ==) | `89.65% <100.00%> (-3.68%)` | :arrow_down: | | [...hadoop/realtime/RealtimeCompactedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lQ29tcGFjdGVkUmVjb3JkUmVhZGVyLmphdmE=) | `77.77% <100.00%> (+0.96%)` | :arrow_up: | | [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `97.67% <100.00%> (+0.11%)` | :arrow_up: | | [.../hadoop/utils/HoodieRealtimeRecordReaderUtils.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZVJlYWx0aW1lUmVjb3JkUmVhZGVyVXRpbHMuamF2YQ==) | `72.03% <100.00%> (+0.23%)` | :arrow_up: | |
[hudi] branch asf-site updated: Travis CI build asf-site
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 09606d3 Travis CI build asf-site 09606d3 is described below commit 09606d31a5252cee3bb05c1a201482feed810c06 Author: CI AuthorDate: Thu Jul 22 04:11:24 2021 + Travis CI build asf-site --- content/docs/writing_data.html | 258 + 1 file changed, 258 insertions(+) diff --git a/content/docs/writing_data.html b/content/docs/writing_data.html index 719cc0d..e1f90e4 100644 --- a/content/docs/writing_data.html +++ b/content/docs/writing_data.html @@ -367,6 +367,7 @@ Syncing to Hive Deletes Optimized DFS Access + Schema Evolution @@ -876,6 +877,263 @@ once created cannot be deleted, but simply expanded as explained before. For workloads with heavy updates, the merge-on-read table provides a nice mechanism for ingesting quickly into smaller files and then later merging them into larger base files via compaction. +Schema Evolution + +Schema evolution is a very important aspect of data management. +Hudi supports common schema evolution scenarios, such as adding a nullable field or promoting a datatype of a field, out-of-the-box. +Furthermore, the evolved schema is queryable across engines, such as Presto, Hive and Spark SQL. +The following table presents a summary of the types of schema changes compatible with different Hudi table types. + + + + + Schema Change + COW + MOR + Remarks + + + + + Add a new nullable column at root level at the end + Yes + Yes + Yes means that a write with evolved schema succeeds and a read following the write succeeds to read entire dataset. + + + Add a new nullable column to inner struct (at the end) + Yes + Yes + + + + Add a new complex type field with default (map and array) + Yes + Yes + + + + Add a new nullable column and change the ordering of fields + No + No + Write succeeds but read fails if the write with evolved schema updated only some of the base files but not all. Currently, Hudi does not maintain a schema registry with history of changes across base files. Nevertheless, if the upsert touched all base files then the read will succeed. + + + Add a custom nullable Hudi meta column, e.g. _hoodie_meta_col + Yes + Yes + + + + Promote datatype from int to long for a field at root level + Yes + Yes + For other types, Hudi supports promotion as specified in http://avro.apache.org/docs/current/spec.html#Schema+Resolution;>Avro schema resolution. + + + Promote datatype from int to long for a nested field + Yes + Yes + + + + Promote datatype from int to long for a complex type (value of map or array) + Yes + Yes + + + + Add a new non-nullable column at root level at the end + No + No + In case of MOR table with Spark data source, write succeeds but read fails. As a workaround, you can make the field nullable. + + + Add a new non-nullable column to inner struct (at the end) + No + No + + + + Change datatype from long to int for a nested field + No + No + + + + Change datatype from long to int for a complex type (value of map or array) + No + No + + + + + +Let us walk through an example to demonstrate the schema evolution support in Hudi. +In the below example, we are going to add a new string field and change the datatype of a field from int to long. + +Welcome to + __ +/ __/__ ___ _/ /__ +_\ \/ _ \/ _ `/ __/ '_/ +/___/ .__/\_,_/_/ /_/\_\ v [...] +/_/ + +Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_292 [...] +Type in expressions to have them evaluated. +Type :help for more information. + +scala import org.apache.hudi.QuickstartUtils._ +import org.apache.hudi.QuickstartUtils._ + +scala import scala.collection.JavaConversions._ +import scala.collection.JavaConversions._ + +scala import org.apache.spark.sql.SaveMode._ +import org.apache.spark.sql.SaveMode._ + +scala import org.apache.hudi.DataSourceReadOptions._ +import org.apache.hudi.DataSourceReadOptions._ + +scala import org.apache.hudi.DataSourceWriteOptions._ +import org.apache.hudi.DataSourceWriteOptions._ + +scala import org.apache.hudi.config.HoodieWriteConfig._ +import org.apache.hudi.config.HoodieWriteConfig._ + +scala import org.apache.spark.sql.types._ +import org.apache.spark.sql.types._ + +scala import org.apache.spark.sql.Row +import org.apache.spark.sql.Row + +scala val tableName =
[GitHub] [hudi] hudi-bot edited a comment on pull request #3325: [WIP] Fixing payload instantiation to include preCombine field in LogRecordScanner
hudi-bot edited a comment on pull request #3325: URL: https://github.com/apache/hudi/pull/3325#issuecomment-884643521 ## CI report: * 7ab74d46bc22b7e93e0ac00a2200b62c4bd120e6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1091) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1548) Fix documentation around schema evolution
[ https://issues.apache.org/jira/browse/HUDI-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385241#comment-17385241 ] ASF GitHub Bot commented on HUDI-1548: -- nsivabalan merged pull request #3257: URL: https://github.com/apache/hudi/pull/3257 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix documentation around schema evolution > -- > > Key: HUDI-1548 > URL: https://issues.apache.org/jira/browse/HUDI-1548 > Project: Apache Hudi > Issue Type: Improvement > Components: Docs >Reporter: sivabalan narayanan >Assignee: Nishith Agarwal >Priority: Blocker > Labels: ', pull-request-available, sev:high, user-support-issues > Fix For: 0.9.0 > > > Clearly call out what kind of schema evolution is supported by hudi in > documentation . > Context: https://github.com/apache/hudi/issues/2331 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] nsivabalan merged pull request #3257: [HUDI-1548] Add documentation for schema evolution
nsivabalan merged pull request #3257: URL: https://github.com/apache/hudi/pull/3257 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[hudi] branch asf-site updated: [HUDI-1548] Add documentation for schema evolution (#3257)
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new ec8bde1 [HUDI-1548] Add documentation for schema evolution (#3257) ec8bde1 is described below commit ec8bde157e7041a93caf3f5f46dab1e5de52f8dd Author: Sagar Sumit AuthorDate: Thu Jul 22 09:39:02 2021 +0530 [HUDI-1548] Add documentation for schema evolution (#3257) --- docs/_docs/2_2_writing_data.md | 189 + 1 file changed, 189 insertions(+) diff --git a/docs/_docs/2_2_writing_data.md b/docs/_docs/2_2_writing_data.md index 86bd14d..00771a3 100644 --- a/docs/_docs/2_2_writing_data.md +++ b/docs/_docs/2_2_writing_data.md @@ -424,3 +424,192 @@ Here are some ways to efficiently manage the storage of your Hudi tables. - Intelligently tuning the [bulk insert parallelism](/docs/configurations.html#withBulkInsertParallelism), can again in nicely sized initial file groups. It is in fact critical to get this right, since the file groups once created cannot be deleted, but simply expanded as explained before. - For workloads with heavy updates, the [merge-on-read table](/docs/concepts.html#merge-on-read-table) provides a nice mechanism for ingesting quickly into smaller files and then later merging them into larger base files via compaction. + + +## Schema Evolution + +Schema evolution is a very important aspect of data management. +Hudi supports common schema evolution scenarios, such as adding a nullable field or promoting a datatype of a field, out-of-the-box. +Furthermore, the evolved schema is queryable across engines, such as Presto, Hive and Spark SQL. +The following table presents a summary of the types of schema changes compatible with different Hudi table types. + +| Schema Change | COW | MOR | Remarks | +| --- | --- | --- | --- | +| Add a new nullable column at root level at the end | Yes | Yes | `Yes` means that a write with evolved schema succeeds and a read following the write succeeds to read entire dataset. | +| Add a new nullable column to inner struct (at the end) | Yes | Yes | +| Add a new complex type field with default (map and array) | Yes | Yes | | +| Add a new nullable column and change the ordering of fields | No | No | Write succeeds but read fails if the write with evolved schema updated only some of the base files but not all. Currently, Hudi does not maintain a schema registry with history of changes across base files. Nevertheless, if the upsert touched all base files then the read will succeed. | +| Add a custom nullable Hudi meta column, e.g. `_hoodie_meta_col` | Yes | Yes | | +| Promote datatype from `int` to `long` for a field at root level | Yes | Yes | For other types, Hudi supports promotion as specified in [Avro schema resolution](http://avro.apache.org/docs/current/spec.html#Schema+Resolution). | +| Promote datatype from `int` to `long` for a nested field | Yes | Yes | +| Promote datatype from `int` to `long` for a complex type (value of map or array) | Yes | Yes | | +| Add a new non-nullable column at root level at the end | No | No | In case of MOR table with Spark data source, write succeeds but read fails. As a **workaround**, you can make the field nullable. | +| Add a new non-nullable column to inner struct (at the end) | No | No | | +| Change datatype from `long` to `int` for a nested field | No | No | | +| Change datatype from `long` to `int` for a complex type (value of map or array) | No | No | | + +Let us walk through an example to demonstrate the schema evolution support in Hudi. +In the below example, we are going to add a new string field and change the datatype of a field from int to long. + +```java +Welcome to + __ +/ __/__ ___ _/ /__ +_\ \/ _ \/ _ `/ __/ '_/ +/___/ .__/\_,_/_/ /_/\_\ version 3.1.2 +/_/ + +Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_292) +Type in expressions to have them evaluated. +Type :help for more information. + +scala> import org.apache.hudi.QuickstartUtils._ +import org.apache.hudi.QuickstartUtils._ + +scala> import scala.collection.JavaConversions._ +import scala.collection.JavaConversions._ + +scala> import org.apache.spark.sql.SaveMode._ +import org.apache.spark.sql.SaveMode._ + +scala> import org.apache.hudi.DataSourceReadOptions._ +import org.apache.hudi.DataSourceReadOptions._ + +scala> import org.apache.hudi.DataSourceWriteOptions._ +import org.apache.hudi.DataSourceWriteOptions._ + +scala> import org.apache.hudi.config.HoodieWriteConfig._ +import org.apache.hudi.config.HoodieWriteConfig._ + +scala> import org.apache.spark.sql.types._ +import org.apache.spark.sql.types._ + +scala> import org.apache.spark.sql.Row +import org.apache.spark.sql.Row + +scala> val tableName = "hudi_trips_cow" +
[GitHub] [hudi] hudi-bot commented on pull request #3325: [WIP] Fixing payload instantiation to include preCombine field in LogRecordScanner
hudi-bot commented on pull request #3325: URL: https://github.com/apache/hudi/pull/3325#issuecomment-884643521 ## CI report: * 7ab74d46bc22b7e93e0ac00a2200b62c4bd120e6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
nsivabalan commented on issue #3323: URL: https://github.com/apache/hudi/issues/3323#issuecomment-884642954 This fix worked for me locally: https://github.com/apache/hudi/pull/3325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan opened a new pull request #3325: [WIP] Fixing payload instantiation to include preCombine field in LogRecordScanner
nsivabalan opened a new pull request #3325: URL: https://github.com/apache/hudi/pull/3325 …Scanner ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
nsivabalan commented on issue #3323: URL: https://github.com/apache/hudi/issues/3323#issuecomment-884640235 @vinothchandar : Do you know the reason why it was designed this way? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2187) Hive integration Improvment
[ https://issues.apache.org/jira/browse/HUDI-2187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385238#comment-17385238 ] ASF GitHub Bot commented on HUDI-2187: -- hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 602b6cdf97b686257bfb17a05d456f9d52f4c147 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1090) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Hive integration Improvment > --- > > Key: HUDI-2187 > URL: https://issues.apache.org/jira/browse/HUDI-2187 > Project: Apache Hudi > Issue Type: Improvement > Components: Hive Integration >Reporter: zhangminglei >Assignee: zhangminglei >Priority: Major > Labels: pull-request-available > > See the details from RFC doc > https://cwiki.apache.org/confluence/display/HUDI/RFC+-+31%3A+Hive+integration+Improvment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3289: [HUDI-2187] Add a shim layer to support multiple hive version
hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 602b6cdf97b686257bfb17a05d456f9d52f4c147 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1090) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig
codecov-commenter edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883347621 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3307](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (5ad4271) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `2.09%`. > The diff coverage is `97.14%`. [](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3307 +/- ## - Coverage 47.74% 45.64% -2.10% - Complexity 5591 5596 +5 Files 938 999 +61 Lines 4182343790+1967 Branches 4213 4403 +190 + Hits 1996819988 +20 - Misses2007022019+1949 + Partials 1785 1783 -2 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `34.55% <ø> (ø)` | | | hudicommon | `48.65% <ø> (+0.01%)` | :arrow_up: | | hudiflink | `59.62% <100.00%> (+0.18%)` | :arrow_up: | | hudihadoopmr | `52.40% <95.65%> (+0.37%)` | :arrow_up: | | hudiintegtest | `0.00% <ø> (?)` | | | hudisparkdatasource | `67.00% <100.00%> (-0.10%)` | :arrow_down: | | hudisync | `55.97% <ø> (ø)` | | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `59.87% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...pache/hudi/hadoop/config/HoodieRealtimeConfig.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL2NvbmZpZy9Ib29kaWVSZWFsdGltZUNvbmZpZy5qYXZh) | `88.88% <88.88%> (+88.88%)` | :arrow_up: | | [...java/org/apache/hudi/table/format/FormatUtils.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9mb3JtYXQvRm9ybWF0VXRpbHMuamF2YQ==) | `89.65% <100.00%> (-3.68%)` | :arrow_down: | | [...hadoop/realtime/RealtimeCompactedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lQ29tcGFjdGVkUmVjb3JkUmVhZGVyLmphdmE=) | `77.77% <100.00%> (+0.96%)` | :arrow_up: | | [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `97.67% <100.00%> (+0.11%)` | :arrow_up: | | [.../hadoop/utils/HoodieRealtimeRecordReaderUtils.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZVJlYWx0aW1lUmVjb3JkUmVhZGVyVXRpbHMuamF2YQ==) | `72.03% <100.00%> (+0.23%)` | :arrow_up: | |
[jira] [Commented] (HUDI-2187) Hive integration Improvment
[ https://issues.apache.org/jira/browse/HUDI-2187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385236#comment-17385236 ] ASF GitHub Bot commented on HUDI-2187: -- hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 1a9c6e49be153e7cf9646da93c1114ae75969c22 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1088) * 602b6cdf97b686257bfb17a05d456f9d52f4c147 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1090) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Hive integration Improvment > --- > > Key: HUDI-2187 > URL: https://issues.apache.org/jira/browse/HUDI-2187 > Project: Apache Hudi > Issue Type: Improvement > Components: Hive Integration >Reporter: zhangminglei >Assignee: zhangminglei >Priority: Major > Labels: pull-request-available > > See the details from RFC doc > https://cwiki.apache.org/confluence/display/HUDI/RFC+-+31%3A+Hive+integration+Improvment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3289: [HUDI-2187] Add a shim layer to support multiple hive version
hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 1a9c6e49be153e7cf9646da93c1114ae75969c22 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1088) * 602b6cdf97b686257bfb17a05d456f9d52f4c147 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1090) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
nsivabalan edited a comment on issue #3323: URL: https://github.com/apache/hudi/issues/3323#issuecomment-884638185 I delved deeper and found the root cause. Could be a bug in code. But surprised how come we have not encountered this so far. When we construct the records back from disk (log blocks), we use [reflection to instantiate the payload](https://github.com/apache/hudi/blob/5a94b6bf54b18739da55ebde10adf93f133e3204/hudi-common/src/main/java/org/apache/hudi/common/util/SpillableMapUtils.java#L116). And we have two constructors with OverwriteWithLatestAvroPayload, one of them takes in ordering field value, while 2nd one does not and assumes natural ordering(sets 0 as preCombine value). Hence when two records are merged, we see the discrepancy. Incase you are wondering, how come snapshot read is giving us correct results, here is the reason. Snapshot read, reads/merges log blocks in reverse and so the latest record always gets picked. Where as w/ incremental, we read/merge log blocks from start to end and so first record gets picked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
nsivabalan commented on issue #3323: URL: https://github.com/apache/hudi/issues/3323#issuecomment-884638185 I delved deeper and found the root cause. Could be a bug in code. But surprised how come we have not encountered this so far. When we construct the records back from disk (log blocks), we use [reflection to instantiate the payload](https://github.com/apache/hudi/blob/5a94b6bf54b18739da55ebde10adf93f133e3204/hudi-common/src/main/java/org/apache/hudi/common/util/SpillableMapUtils.java#L116). And we have two constructors with OverwriteWithLatestAvroPayload, one of them takes in ordering field value, while 2nd one does not and assumes natural ordering(sets 0 as preCombine value). Hence when two records are merged, we see the discrepancy. Incase you are wondering, how come snapshot read is giving us correct results, here is the reason. Snapshot reads log blocks in reverse and so the latest record always gets picked. Where as w/ incremental, we read log blocks from start to end and so first record gets picked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385235#comment-17385235 ] ASF GitHub Bot commented on HUDI-1771: -- hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 4e731ff162b777b85e6c6c09e5387a2f0215197b Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1087) * a46c6f21415c65e005ba9ac267cd2bda4528a39b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1089) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Propagate CDC format for hoodie > --- > > Key: HUDI-1771 > URL: https://issues.apache.org/jira/browse/HUDI-1771 > Project: Apache Hudi > Issue Type: New Feature > Components: Flink Integration >Reporter: Danny Chen >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available, sev:normal > Fix For: 0.9.0 > > > Like what we discussed in the dev mailing list: > https://lists.apache.org/thread.html/r31b2d1404e4e043a5f875b78105ba6f9a801e78f265ad91242ad5eb2%40%3Cdev.hudi.apache.org%3E > Keep the change flags make new use cases possible: using HUDI as the unified > storage format for DWD and DWS layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie
hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 4e731ff162b777b85e6c6c09e5387a2f0215197b Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1087) * a46c6f21415c65e005ba9ac267cd2bda4528a39b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1089) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HUDI-2188) Improve test for the insert_overwrite and insert_overwrite_table in hoodieDeltaStreamer
[ https://issues.apache.org/jira/browse/HUDI-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samrat Deb reassigned HUDI-2188: Assignee: Samrat Deb > Improve test for the insert_overwrite and insert_overwrite_table in > hoodieDeltaStreamer > --- > > Key: HUDI-2188 > URL: https://issues.apache.org/jira/browse/HUDI-2188 > Project: Apache Hudi > Issue Type: Test >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > > InsertOverwrite overwrites only the partitions matching the incoming records. > need to add a test that verifies insert_overwrite does not overwrite > mismatched partitions. > reference -: https://github.com/apache/hudi/pull/3184/files#r670993094 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig
codecov-commenter edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883347621 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3307](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (5ad4271) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `2.09%`. > The diff coverage is `97.14%`. [](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3307 +/- ## - Coverage 47.74% 45.64% -2.10% - Complexity 5591 5596 +5 Files 938 999 +61 Lines 4182343790+1967 Branches 4213 4403 +190 + Hits 1996819988 +20 - Misses2007022019+1949 + Partials 1785 1783 -2 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `34.55% <ø> (ø)` | | | hudicommon | `48.65% <ø> (+0.01%)` | :arrow_up: | | hudiflink | `59.62% <100.00%> (+0.18%)` | :arrow_up: | | hudihadoopmr | `52.40% <95.65%> (+0.37%)` | :arrow_up: | | hudiintegtest | `0.00% <ø> (?)` | | | hudisparkdatasource | `67.00% <100.00%> (-0.10%)` | :arrow_down: | | hudisync | `55.97% <ø> (ø)` | | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `59.87% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...pache/hudi/hadoop/config/HoodieRealtimeConfig.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL2NvbmZpZy9Ib29kaWVSZWFsdGltZUNvbmZpZy5qYXZh) | `88.88% <88.88%> (+88.88%)` | :arrow_up: | | [...java/org/apache/hudi/table/format/FormatUtils.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9mb3JtYXQvRm9ybWF0VXRpbHMuamF2YQ==) | `89.65% <100.00%> (-3.68%)` | :arrow_down: | | [...hadoop/realtime/RealtimeCompactedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lQ29tcGFjdGVkUmVjb3JkUmVhZGVyLmphdmE=) | `77.77% <100.00%> (+0.96%)` | :arrow_up: | | [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `97.67% <100.00%> (+0.11%)` | :arrow_up: | | [.../hadoop/utils/HoodieRealtimeRecordReaderUtils.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZVJlYWx0aW1lUmVjb3JkUmVhZGVyVXRpbHMuamF2YQ==) | `72.03% <100.00%> (+0.23%)` | :arrow_up: | |
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385233#comment-17385233 ] ASF GitHub Bot commented on HUDI-1771: -- hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 4e731ff162b777b85e6c6c09e5387a2f0215197b Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1087) * a46c6f21415c65e005ba9ac267cd2bda4528a39b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Propagate CDC format for hoodie > --- > > Key: HUDI-1771 > URL: https://issues.apache.org/jira/browse/HUDI-1771 > Project: Apache Hudi > Issue Type: New Feature > Components: Flink Integration >Reporter: Danny Chen >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available, sev:normal > Fix For: 0.9.0 > > > Like what we discussed in the dev mailing list: > https://lists.apache.org/thread.html/r31b2d1404e4e043a5f875b78105ba6f9a801e78f265ad91242ad5eb2%40%3Cdev.hudi.apache.org%3E > Keep the change flags make new use cases possible: using HUDI as the unified > storage format for DWD and DWS layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie
hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 4e731ff162b777b85e6c6c09e5387a2f0215197b Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1087) * a46c6f21415c65e005ba9ac267cd2bda4528a39b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385230#comment-17385230 ] ASF GitHub Bot commented on HUDI-1771: -- hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 823eea932ca572fd4fdba011ab74dc2b52f277b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1077) * 4e731ff162b777b85e6c6c09e5387a2f0215197b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1087) * a46c6f21415c65e005ba9ac267cd2bda4528a39b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Propagate CDC format for hoodie > --- > > Key: HUDI-1771 > URL: https://issues.apache.org/jira/browse/HUDI-1771 > Project: Apache Hudi > Issue Type: New Feature > Components: Flink Integration >Reporter: Danny Chen >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available, sev:normal > Fix For: 0.9.0 > > > Like what we discussed in the dev mailing list: > https://lists.apache.org/thread.html/r31b2d1404e4e043a5f875b78105ba6f9a801e78f265ad91242ad5eb2%40%3Cdev.hudi.apache.org%3E > Keep the change flags make new use cases possible: using HUDI as the unified > storage format for DWD and DWS layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie
hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 823eea932ca572fd4fdba011ab74dc2b52f277b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1077) * 4e731ff162b777b85e6c6c09e5387a2f0215197b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1087) * a46c6f21415c65e005ba9ac267cd2bda4528a39b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385226#comment-17385226 ] ASF GitHub Bot commented on HUDI-2176: -- codecov-commenter edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883054849 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3306](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (719bb10) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `19.98%`. > The diff coverage is `24.65%`. [](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3306 +/- ## = - Coverage 47.74% 27.76% -19.99% + Complexity 5591 1330 -4261 = Files 938 386 -552 Lines 4182315582-26241 Branches 4213 1390 -2823 = - Hits 19968 4326-15642 + Misses2007010932 -9138 + Partials 1785 324 -1461 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.33% <24.65%> (-13.23%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.87% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/io/HoodieKeyLocationFetchHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUtleUxvY2F0aW9uRmV0Y2hIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/io/HoodieSortedMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVNvcnRlZE1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table
codecov-commenter edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883054849 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3306](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (719bb10) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `19.98%`. > The diff coverage is `24.65%`. [](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3306 +/- ## = - Coverage 47.74% 27.76% -19.99% + Complexity 5591 1330 -4261 = Files 938 386 -552 Lines 4182315582-26241 Branches 4213 1390 -2823 = - Hits 19968 4326-15642 + Misses2007010932 -9138 + Partials 1785 324 -1461 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.33% <24.65%> (-13.23%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.87% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/io/HoodieKeyLocationFetchHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUtleUxvY2F0aW9uRmV0Y2hIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/io/HoodieSortedMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVNvcnRlZE1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | |
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig
codecov-commenter edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883347621 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3307](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (5ad4271) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `2.09%`. > The diff coverage is `97.14%`. [](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3307 +/- ## - Coverage 47.74% 45.64% -2.10% - Complexity 5591 5596 +5 Files 938 999 +61 Lines 4182343790+1967 Branches 4213 4403 +190 + Hits 1996819988 +20 - Misses2007022019+1949 + Partials 1785 1783 -2 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `34.55% <ø> (ø)` | | | hudicommon | `48.65% <ø> (+0.01%)` | :arrow_up: | | hudiflink | `59.62% <100.00%> (+0.18%)` | :arrow_up: | | hudihadoopmr | `52.40% <95.65%> (+0.37%)` | :arrow_up: | | hudiintegtest | `0.00% <ø> (?)` | | | hudisparkdatasource | `67.00% <100.00%> (-0.10%)` | :arrow_down: | | hudisync | `55.97% <ø> (ø)` | | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `59.87% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3307?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...pache/hudi/hadoop/config/HoodieRealtimeConfig.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL2NvbmZpZy9Ib29kaWVSZWFsdGltZUNvbmZpZy5qYXZh) | `88.88% <88.88%> (+88.88%)` | :arrow_up: | | [...java/org/apache/hudi/table/format/FormatUtils.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1mbGluay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS90YWJsZS9mb3JtYXQvRm9ybWF0VXRpbHMuamF2YQ==) | `89.65% <100.00%> (-3.68%)` | :arrow_down: | | [...hadoop/realtime/RealtimeCompactedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lQ29tcGFjdGVkUmVjb3JkUmVhZGVyLmphdmE=) | `77.77% <100.00%> (+0.96%)` | :arrow_up: | | [.../hadoop/realtime/RealtimeUnmergedRecordReader.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3JlYWx0aW1lL1JlYWx0aW1lVW5tZXJnZWRSZWNvcmRSZWFkZXIuamF2YQ==) | `97.67% <100.00%> (+0.11%)` | :arrow_up: | | [.../hadoop/utils/HoodieRealtimeRecordReaderUtils.java](https://codecov.io/gh/apache/hudi/pull/3307/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL3V0aWxzL0hvb2RpZVJlYWx0aW1lUmVjb3JkUmVhZGVyVXRpbHMuamF2YQ==) | `72.03% <100.00%> (+0.23%)` | :arrow_up: | |
[jira] [Commented] (HUDI-2191) Bump flink version to 1.13.1
[ https://issues.apache.org/jira/browse/HUDI-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385223#comment-17385223 ] ASF GitHub Bot commented on HUDI-2191: -- garyli1019 commented on pull request #3291: URL: https://github.com/apache/hudi/pull/3291#issuecomment-884627266 @danny0405 What's the impact of upgrading from 1.12.2 to 1.12.3? we should be very cautious about the version upgrade unless they are fully backward compatible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Bump flink version to 1.13.1 > > > Key: HUDI-2191 > URL: https://issues.apache.org/jira/browse/HUDI-2191 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Aims to use flink 1.13.1 for 0.9.0 release. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] garyli1019 commented on pull request #3291: [HUDI-2191] Bump flink version to 1.13.1
garyli1019 commented on pull request #3291: URL: https://github.com/apache/hudi/pull/3291#issuecomment-884627266 @danny0405 What's the impact of upgrading from 1.12.2 to 1.12.3? we should be very cautious about the version upgrade unless they are fully backward compatible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2191) Bump flink version to 1.13.1
[ https://issues.apache.org/jira/browse/HUDI-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385219#comment-17385219 ] ASF GitHub Bot commented on HUDI-2191: -- danny0405 commented on pull request #3291: URL: https://github.com/apache/hudi/pull/3291#issuecomment-884625586 > @danny0405 I think we should at least support 1.12 for the 0.9.0 release, if you think this upgrade is necessary, I'd recommend support both 1.12 and 1.13. Maybe we can have a V1 and V2 connector? I'm planning to upgrade the version to 1.12.3 for 0.9.0 release and after the release upgrade it to 1.13.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Bump flink version to 1.13.1 > > > Key: HUDI-2191 > URL: https://issues.apache.org/jira/browse/HUDI-2191 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Aims to use flink 1.13.1 for 0.9.0 release. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] danny0405 commented on pull request #3291: [HUDI-2191] Bump flink version to 1.13.1
danny0405 commented on pull request #3291: URL: https://github.com/apache/hudi/pull/3291#issuecomment-884625586 > @danny0405 I think we should at least support 1.12 for the 0.9.0 release, if you think this upgrade is necessary, I'd recommend support both 1.12 and 1.13. Maybe we can have a V1 and V2 connector? I'm planning to upgrade the version to 1.12.3 for 0.9.0 release and after the release upgrade it to 1.13.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2187) Hive integration Improvment
[ https://issues.apache.org/jira/browse/HUDI-2187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385214#comment-17385214 ] ASF GitHub Bot commented on HUDI-2187: -- hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 1a9c6e49be153e7cf9646da93c1114ae75969c22 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1088) * 602b6cdf97b686257bfb17a05d456f9d52f4c147 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Hive integration Improvment > --- > > Key: HUDI-2187 > URL: https://issues.apache.org/jira/browse/HUDI-2187 > Project: Apache Hudi > Issue Type: Improvement > Components: Hive Integration >Reporter: zhangminglei >Assignee: zhangminglei >Priority: Major > Labels: pull-request-available > > See the details from RFC doc > https://cwiki.apache.org/confluence/display/HUDI/RFC+-+31%3A+Hive+integration+Improvment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385215#comment-17385215 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883052706 ## CI report: * 719bb1046e9d69d88395d65d2f928af35fb9d9bd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1086) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virutal keys support for COW all operations > --- > > Key: HUDI-2176 > URL: https://issues.apache.org/jira/browse/HUDI-2176 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virutal keys support for COW all operations > (merge handle) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table
hudi-bot edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883052706 ## CI report: * 719bb1046e9d69d88395d65d2f928af35fb9d9bd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1086) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3289: [HUDI-2187] Add a shim layer to support multiple hive version
hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 1a9c6e49be153e7cf9646da93c1114ae75969c22 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1088) * 602b6cdf97b686257bfb17a05d456f9d52f4c147 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2187) Hive integration Improvment
[ https://issues.apache.org/jira/browse/HUDI-2187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385212#comment-17385212 ] ASF GitHub Bot commented on HUDI-2187: -- hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 3df6f8079655b7128a0abc9b362133396f7c89e8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1082) * 1a9c6e49be153e7cf9646da93c1114ae75969c22 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1088) * 602b6cdf97b686257bfb17a05d456f9d52f4c147 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Hive integration Improvment > --- > > Key: HUDI-2187 > URL: https://issues.apache.org/jira/browse/HUDI-2187 > Project: Apache Hudi > Issue Type: Improvement > Components: Hive Integration >Reporter: zhangminglei >Assignee: zhangminglei >Priority: Major > Labels: pull-request-available > > See the details from RFC doc > https://cwiki.apache.org/confluence/display/HUDI/RFC+-+31%3A+Hive+integration+Improvment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3289: [HUDI-2187] Add a shim layer to support multiple hive version
hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 3df6f8079655b7128a0abc9b362133396f7c89e8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1082) * 1a9c6e49be153e7cf9646da93c1114ae75969c22 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1088) * 602b6cdf97b686257bfb17a05d456f9d52f4c147 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2187) Hive integration Improvment
[ https://issues.apache.org/jira/browse/HUDI-2187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385211#comment-17385211 ] ASF GitHub Bot commented on HUDI-2187: -- hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 3df6f8079655b7128a0abc9b362133396f7c89e8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1082) * 1a9c6e49be153e7cf9646da93c1114ae75969c22 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1088) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Hive integration Improvment > --- > > Key: HUDI-2187 > URL: https://issues.apache.org/jira/browse/HUDI-2187 > Project: Apache Hudi > Issue Type: Improvement > Components: Hive Integration >Reporter: zhangminglei >Assignee: zhangminglei >Priority: Major > Labels: pull-request-available > > See the details from RFC doc > https://cwiki.apache.org/confluence/display/HUDI/RFC+-+31%3A+Hive+integration+Improvment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3289: [HUDI-2187] Add a shim layer to support multiple hive version
hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 3df6f8079655b7128a0abc9b362133396f7c89e8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1082) * 1a9c6e49be153e7cf9646da93c1114ae75969c22 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1088) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385209#comment-17385209 ] ASF GitHub Bot commented on HUDI-2176: -- codecov-commenter edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883054849 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3306](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (719bb10) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `31.61%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3306 +/- ## = - Coverage 47.74% 16.12% -31.62% + Complexity 5591 505 -5086 = Files 938 280 -658 Lines 4182311856-29967 Branches 4213 989 -3224 = - Hits 19968 1912-18056 + Misses20070 9780-10290 + Partials 1785 164 -1621 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.87% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/io/HoodieKeyLocationFetchHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUtleUxvY2F0aW9uRmV0Y2hIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/io/HoodieSortedMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVNvcnRlZE1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | |
[jira] [Commented] (HUDI-2187) Hive integration Improvment
[ https://issues.apache.org/jira/browse/HUDI-2187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385207#comment-17385207 ] ASF GitHub Bot commented on HUDI-2187: -- hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 3df6f8079655b7128a0abc9b362133396f7c89e8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1082) * 1a9c6e49be153e7cf9646da93c1114ae75969c22 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Hive integration Improvment > --- > > Key: HUDI-2187 > URL: https://issues.apache.org/jira/browse/HUDI-2187 > Project: Apache Hudi > Issue Type: Improvement > Components: Hive Integration >Reporter: zhangminglei >Assignee: zhangminglei >Priority: Major > Labels: pull-request-available > > See the details from RFC doc > https://cwiki.apache.org/confluence/display/HUDI/RFC+-+31%3A+Hive+integration+Improvment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table
codecov-commenter edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883054849 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3306](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (719bb10) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `31.61%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3306 +/- ## = - Coverage 47.74% 16.12% -31.62% + Complexity 5591 505 -5086 = Files 938 280 -658 Lines 4182311856-29967 Branches 4213 989 -3224 = - Hits 19968 1912-18056 + Misses20070 9780-10290 + Partials 1785 164 -1621 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.87% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/io/HoodieKeyLocationFetchHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUtleUxvY2F0aW9uRmV0Y2hIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/io/HoodieSortedMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVNvcnRlZE1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | |
[GitHub] [hudi] hudi-bot edited a comment on pull request #3289: [HUDI-2187] Add a shim layer to support multiple hive version
hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 3df6f8079655b7128a0abc9b362133396f7c89e8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1082) * 1a9c6e49be153e7cf9646da93c1114ae75969c22 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2191) Bump flink version to 1.13.1
[ https://issues.apache.org/jira/browse/HUDI-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385205#comment-17385205 ] ASF GitHub Bot commented on HUDI-2191: -- garyli1019 commented on pull request #3291: URL: https://github.com/apache/hudi/pull/3291#issuecomment-884619431 @danny0405 I think we should at least support 1.12 for the 0.9.0 release, if you think this upgrade is necessary, I'd recommend support both 1.12 and 1.13. Maybe we can have a V1 and V2 connector? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Bump flink version to 1.13.1 > > > Key: HUDI-2191 > URL: https://issues.apache.org/jira/browse/HUDI-2191 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Aims to use flink 1.13.1 for 0.9.0 release. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] garyli1019 commented on pull request #3291: [HUDI-2191] Bump flink version to 1.13.1
garyli1019 commented on pull request #3291: URL: https://github.com/apache/hudi/pull/3291#issuecomment-884619431 @danny0405 I think we should at least support 1.12 for the 0.9.0 release, if you think this upgrade is necessary, I'd recommend support both 1.12 and 1.13. Maybe we can have a V1 and V2 connector? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
nsivabalan commented on issue #3323: URL: https://github.com/apache/hudi/issues/3323#issuecomment-884618704 my bad. looks like you do have a high preCombine value. Just that the datatype is different from what I have tried. may I know what was the datatype of "ts" column in hudi table ? I am looking to reproduce w/ spark shell. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385202#comment-17385202 ] ASF GitHub Bot commented on HUDI-1771: -- hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 823eea932ca572fd4fdba011ab74dc2b52f277b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1077) * 4e731ff162b777b85e6c6c09e5387a2f0215197b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1087) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Propagate CDC format for hoodie > --- > > Key: HUDI-1771 > URL: https://issues.apache.org/jira/browse/HUDI-1771 > Project: Apache Hudi > Issue Type: New Feature > Components: Flink Integration >Reporter: Danny Chen >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available, sev:normal > Fix For: 0.9.0 > > > Like what we discussed in the dev mailing list: > https://lists.apache.org/thread.html/r31b2d1404e4e043a5f875b78105ba6f9a801e78f265ad91242ad5eb2%40%3Cdev.hudi.apache.org%3E > Keep the change flags make new use cases possible: using HUDI as the unified > storage format for DWD and DWS layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie
hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 823eea932ca572fd4fdba011ab74dc2b52f277b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1077) * 4e731ff162b777b85e6c6c09e5387a2f0215197b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1087) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385201#comment-17385201 ] ASF GitHub Bot commented on HUDI-1771: -- hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 823eea932ca572fd4fdba011ab74dc2b52f277b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1077) * 4e731ff162b777b85e6c6c09e5387a2f0215197b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Propagate CDC format for hoodie > --- > > Key: HUDI-1771 > URL: https://issues.apache.org/jira/browse/HUDI-1771 > Project: Apache Hudi > Issue Type: New Feature > Components: Flink Integration >Reporter: Danny Chen >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available, sev:normal > Fix For: 0.9.0 > > > Like what we discussed in the dev mailing list: > https://lists.apache.org/thread.html/r31b2d1404e4e043a5f875b78105ba6f9a801e78f265ad91242ad5eb2%40%3Cdev.hudi.apache.org%3E > Keep the change flags make new use cases possible: using HUDI as the unified > storage format for DWD and DWS layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie
hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 823eea932ca572fd4fdba011ab74dc2b52f277b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1077) * 4e731ff162b777b85e6c6c09e5387a2f0215197b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
nsivabalan commented on issue #3323: URL: https://github.com/apache/hudi/issues/3323#issuecomment-884616158 Hudi uses preCombine to merge records from multiple log blocks fyi, but uses combineAndGetUpdateValue() when merging base file w/ the merged records from log. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries
nsivabalan commented on issue #3323:
URL: https://github.com/apache/hudi/issues/3323#issuecomment-884615732
I guess its bcoz, you don't have a higher preCombine value for new commits
made. I see that all your preCombine is set to same value. Can you try it out
and let me know.
my local trail:
https://gist.github.com/nsivabalan/d75f3f948c5f01a0b3454b986c3b9fe6
output:
```
scala> spark.sql("select rowId, preComb, value from
hudi_trips_incremental").show(false)
+-+---+-+
|rowId|preComb|value|
+-+---+-+
|row_3|2 |2|
|row_2|2 |2|
|row_1|2 |2|
|row_4|2 |2|
|row_5|2 |2|
|row_6|2 |2|
|row_7|2 |2|
|row_8|2 |2|
|row_9|2 |2|
+-+---+-+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] leoluan2009 closed pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig
leoluan2009 closed pull request #3307: URL: https://github.com/apache/hudi/pull/3307 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2205) Rollback inflight compaction for flink writer
[ https://issues.apache.org/jira/browse/HUDI-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385196#comment-17385196 ] ASF GitHub Bot commented on HUDI-2205: -- danny0405 closed pull request #3320: URL: https://github.com/apache/hudi/pull/3320 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Rollback inflight compaction for flink writer > - > > Key: HUDI-2205 > URL: https://issues.apache.org/jira/browse/HUDI-2205 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385199#comment-17385199 ] ASF GitHub Bot commented on HUDI-2176: -- codecov-commenter edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883054849 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3306](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (719bb10) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `44.91%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3306 +/- ## - Coverage 47.74% 2.82% -44.92% + Complexity 5591 85 -5506 Files 938 280 -658 Lines 41823 11856-29967 Branches 4213 989 -3224 - Hits 19968 335-19633 + Misses20070 11495 -8575 + Partials 1785 26 -1759 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.97% <ø> (-50.91%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/io/HoodieKeyLocationFetchHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUtleUxvY2F0aW9uRmV0Y2hIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/io/HoodieSortedMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVNvcnRlZE1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | |
[jira] [Commented] (HUDI-2205) Rollback inflight compaction for flink writer
[ https://issues.apache.org/jira/browse/HUDI-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385197#comment-17385197 ] ASF GitHub Bot commented on HUDI-2205: -- danny0405 opened a new pull request #3320: URL: https://github.com/apache/hudi/pull/3320 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Rollback inflight compaction for flink writer > - > > Key: HUDI-2205 > URL: https://issues.apache.org/jira/browse/HUDI-2205 > Project: Apache Hudi > Issue Type: Improvement > Components: Flink Integration >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] codecov-commenter edited a comment on pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table
codecov-commenter edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883054849 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report > Merging [#3306](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (719bb10) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `44.91%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master #3306 +/- ## - Coverage 47.74% 2.82% -44.92% + Complexity 5591 85 -5506 Files 938 280 -658 Lines 41823 11856-29967 Branches 4213 989 -3224 - Hits 19968 335-19633 + Misses20070 11495 -8575 + Partials 1785 26 -1759 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `8.97% <ø> (-50.91%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3306?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...g/apache/hudi/io/HoodieKeyLocationFetchHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUtleUxvY2F0aW9uRmV0Y2hIYW5kbGUuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...ain/java/org/apache/hudi/io/HoodieMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...va/org/apache/hudi/io/HoodieSortedMergeHandle.java](https://codecov.io/gh/apache/hudi/pull/3306/diff?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZVNvcnRlZE1lcmdlSGFuZGxlLmphdmE=) | `0.00% <0.00%> (ø)` | | |
[GitHub] [hudi] danny0405 closed pull request #3320: [HUDI-2205] Rollback inflight compaction for flink writer
danny0405 closed pull request #3320: URL: https://github.com/apache/hudi/pull/3320 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[
https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385193#comment-17385193
]
ASF GitHub Bot commented on HUDI-1771:
--
danny0405 commented on a change in pull request #3285:
URL: https://github.com/apache/hudi/pull/3285#discussion_r674454632
##
File path:
hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadInputFormat.java
##
@@ -615,24 +621,25 @@ public boolean reachedEnd() throws IOException {
while (logKeysIterator.hasNext()) {
final String curKey = logKeysIterator.next();
if (!keyToSkip.contains(curKey)) {
- Option insertAvroRecord = getInsetValue(curKey);
+ final HoodieRecord record = logRecords.get(curKey);
+ Option insertAvroRecord = getInsetValue(record);
if (insertAvroRecord.isPresent()) {
// the record is a DELETE if insertAvroRecord not present, skipping
-GenericRecord requiredAvroRecord = buildAvroRecordBySchema(
+GenericRecord avroRecord = buildAvroRecordBySchema(
insertAvroRecord.get(),
requiredSchema,
requiredPos,
recordBuilder);
-this.currentRecord = (RowData)
avroToRowDataConverter.convert(requiredAvroRecord);
+this.currentRecord = (RowData)
avroToRowDataConverter.convert(avroRecord);
+
this.currentRecord.setRowKind(FormatUtils.getRowKind(insertAvroRecord.get(),
this.operationPos));
return false;
}
}
}
return true;
}
-private Option getInsetValue(String curKey) throws
IOException {
- final HoodieRecord record = logRecords.get(curKey);
+private Option getInsetValue(HoodieRecord record) throws
IOException {
Review comment:
Seems unnecessary change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Propagate CDC format for hoodie
> ---
>
> Key: HUDI-1771
> URL: https://issues.apache.org/jira/browse/HUDI-1771
> Project: Apache Hudi
> Issue Type: New Feature
> Components: Flink Integration
>Reporter: Danny Chen
>Assignee: Zheng yunhong
>Priority: Major
> Labels: pull-request-available, sev:normal
> Fix For: 0.9.0
>
>
> Like what we discussed in the dev mailing list:
> https://lists.apache.org/thread.html/r31b2d1404e4e043a5f875b78105ba6f9a801e78f265ad91242ad5eb2%40%3Cdev.hudi.apache.org%3E
> Keep the change flags make new use cases possible: using HUDI as the unified
> storage format for DWD and DWS layer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] danny0405 commented on a change in pull request #3285: [HUDI-1771] Propagate CDC format for hoodie
danny0405 commented on a change in pull request #3285:
URL: https://github.com/apache/hudi/pull/3285#discussion_r674454632
##
File path:
hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadInputFormat.java
##
@@ -615,24 +621,25 @@ public boolean reachedEnd() throws IOException {
while (logKeysIterator.hasNext()) {
final String curKey = logKeysIterator.next();
if (!keyToSkip.contains(curKey)) {
- Option insertAvroRecord = getInsetValue(curKey);
+ final HoodieRecord record = logRecords.get(curKey);
+ Option insertAvroRecord = getInsetValue(record);
if (insertAvroRecord.isPresent()) {
// the record is a DELETE if insertAvroRecord not present, skipping
-GenericRecord requiredAvroRecord = buildAvroRecordBySchema(
+GenericRecord avroRecord = buildAvroRecordBySchema(
insertAvroRecord.get(),
requiredSchema,
requiredPos,
recordBuilder);
-this.currentRecord = (RowData)
avroToRowDataConverter.convert(requiredAvroRecord);
+this.currentRecord = (RowData)
avroToRowDataConverter.convert(avroRecord);
+
this.currentRecord.setRowKind(FormatUtils.getRowKind(insertAvroRecord.get(),
this.operationPos));
return false;
}
}
}
return true;
}
-private Option getInsetValue(String curKey) throws
IOException {
- final HoodieRecord record = logRecords.get(curKey);
+private Option getInsetValue(HoodieRecord record) throws
IOException {
Review comment:
Seems unnecessary change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-2195) Sync Hive Failed When Execute CTAS In Spark2 And Spark3
[
https://issues.apache.org/jira/browse/HUDI-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385191#comment-17385191
]
ASF GitHub Bot commented on HUDI-2195:
--
pengzhiwei2018 commented on a change in pull request #3299:
URL: https://github.com/apache/hudi/pull/3299#discussion_r674453778
##
File path: packaging/hudi-spark-bundle/pom.xml
##
@@ -367,7 +369,18 @@
curator-recipes
${zk-curator.version}
-
+
+
+ org.json
+ json
+ 20200518
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Sync Hive Failed When Execute CTAS In Spark2 And Spark3
>
>
> Key: HUDI-2195
> URL: https://issues.apache.org/jira/browse/HUDI-2195
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available
>
> When execute CTAS in spark2, the follow exception will throw out:
> {code:java}
> java.lang.NoClassDefFoundError: org/json/JSONException
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:10847)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10047)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10128)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:209)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> {code}
> While executing CTAS in spark3, the follow exception throw out:
> {code:java}
> java.lang.NoClassDefFoundError:
> org/apache/calcite/rel/type/RelDataTypeSystemjava.lang.NoClassDefFoundError:
> org/apache/calcite/rel/type/RelDataTypeSystem at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzerFactory.get(SemanticAnalyzerFactory.java:318)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:484) at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317) at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227) at
> org.apache.hudi.hive.HoodieHiveClient.updateHiveSQLs(HoodieHiveClient.java:458)
> at
> org.apache.hudi.hive.HoodieHiveClient.updateHiveSQLUsingHiveDriver(HoodieHiveClient.java:448)
> at
> org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:426)
> at
> org.apache.hudi.hive.HoodieHiveClient.createTable(HoodieHiveClient.java:322)
> at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:234) at
> org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:179) at
> org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:130)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-2195) Sync Hive Failed When Execute CTAS In Spark2 And Spark3
[
https://issues.apache.org/jira/browse/HUDI-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385192#comment-17385192
]
ASF GitHub Bot commented on HUDI-2195:
--
pengzhiwei2018 commented on a change in pull request #3299:
URL: https://github.com/apache/hudi/pull/3299#discussion_r674453822
##
File path: packaging/hudi-spark-bundle/pom.xml
##
@@ -367,7 +369,18 @@
curator-recipes
${zk-curator.version}
-
+
+
+ org.json
+ json
+ 20200518
+
+
+
+ org.apache.calcite
+ calcite-core
+ 1.16.0
Review comment:
done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Sync Hive Failed When Execute CTAS In Spark2 And Spark3
>
>
> Key: HUDI-2195
> URL: https://issues.apache.org/jira/browse/HUDI-2195
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available
>
> When execute CTAS in spark2, the follow exception will throw out:
> {code:java}
> java.lang.NoClassDefFoundError: org/json/JSONException
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:10847)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10047)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10128)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:209)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> {code}
> While executing CTAS in spark3, the follow exception throw out:
> {code:java}
> java.lang.NoClassDefFoundError:
> org/apache/calcite/rel/type/RelDataTypeSystemjava.lang.NoClassDefFoundError:
> org/apache/calcite/rel/type/RelDataTypeSystem at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzerFactory.get(SemanticAnalyzerFactory.java:318)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:484) at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317) at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227) at
> org.apache.hudi.hive.HoodieHiveClient.updateHiveSQLs(HoodieHiveClient.java:458)
> at
> org.apache.hudi.hive.HoodieHiveClient.updateHiveSQLUsingHiveDriver(HoodieHiveClient.java:448)
> at
> org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:426)
> at
> org.apache.hudi.hive.HoodieHiveClient.createTable(HoodieHiveClient.java:322)
> at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:234) at
> org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:179) at
> org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:130)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3299: [HUDI-2195] Sync Hive Failed When Execute CTAS In Spark2 And Spark3
pengzhiwei2018 commented on a change in pull request #3299:
URL: https://github.com/apache/hudi/pull/3299#discussion_r674453822
##
File path: packaging/hudi-spark-bundle/pom.xml
##
@@ -367,7 +369,18 @@
curator-recipes
${zk-curator.version}
-
+
+
+ org.json
+ json
+ 20200518
+
+
+
+ org.apache.calcite
+ calcite-core
+ 1.16.0
Review comment:
done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3299: [HUDI-2195] Sync Hive Failed When Execute CTAS In Spark2 And Spark3
pengzhiwei2018 commented on a change in pull request #3299:
URL: https://github.com/apache/hudi/pull/3299#discussion_r674453778
##
File path: packaging/hudi-spark-bundle/pom.xml
##
@@ -367,7 +369,18 @@
curator-recipes
${zk-curator.version}
-
+
+
+ org.json
+ json
+ 20200518
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3236: why use avro not parquet in MOR
nsivabalan commented on issue #3236: URL: https://github.com/apache/hudi/issues/3236#issuecomment-884608912 Let us know if you have any more questions. if not, will close the ticket. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3246: OCC correct problem
nsivabalan commented on issue #3246: URL: https://github.com/apache/hudi/issues/3246#issuecomment-884608726 yes, this is a known limitation I guess. @n3nash : Can you confirm and add more if any. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3321: [SUPPORT] Setting _hoodie_is_deleted column is not deleting records when using Spark DataSource.
nsivabalan commented on issue #3321: URL: https://github.com/apache/hudi/issues/3321#issuecomment-884608382 I see that you are using bulk_insert operation. Don't think w/ bulk_insert we honor the _hoodie_is_deleted. Can you try setting your operation to "upsert". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3321: [SUPPORT] Setting _hoodie_is_deleted column is not deleting records when using Spark DataSource.
nsivabalan commented on issue #3321: URL: https://github.com/apache/hudi/issues/3321#issuecomment-884607348 May I know whats the schema of your table. do you set default value for "_hoodie_is_deleted" as false? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385182#comment-17385182 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883052706 ## CI report: * 9fd68774b3721b403bba916fec6b55a3a2bba7ba Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1084) * 719bb1046e9d69d88395d65d2f928af35fb9d9bd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1086) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virutal keys support for COW all operations > --- > > Key: HUDI-2176 > URL: https://issues.apache.org/jira/browse/HUDI-2176 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virutal keys support for COW all operations > (merge handle) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table
hudi-bot edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883052706 ## CI report: * 9fd68774b3721b403bba916fec6b55a3a2bba7ba Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1084) * 719bb1046e9d69d88395d65d2f928af35fb9d9bd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1086) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
