[GitHub] [hudi] lw309637554 commented on a change in pull request #2776: [HUDI-1768] spark datasource support schema validate add column

2021-04-06 Thread GitBox
lw309637554 commented on a change in pull request #2776: URL: https://github.com/apache/hudi/pull/2776#discussion_r608359776 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala ## @@ -669,4 +669,90 @@ class

[GitHub] [hudi] lw309637554 commented on a change in pull request #2776: [HUDI-1768] spark datasource support schema validate add column

2021-04-06 Thread GitBox
lw309637554 commented on a change in pull request #2776: URL: https://github.com/apache/hudi/pull/2776#discussion_r608359051 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala ## @@ -669,4 +669,90 @@ class

[GitHub] [hudi] yanghua commented on a change in pull request #2740: [HUDI-1055] Remove hardcoded parquet in tests

2021-04-06 Thread GitBox
yanghua commented on a change in pull request #2740: URL: https://github.com/apache/hudi/pull/2740#discussion_r608299680 ## File path: hudi-cli/src/main/scala/org/apache/hudi/cli/SparkHelpers.scala ## @@ -40,7 +40,7 @@ import scala.collection.mutable._ object SparkHelpers {

[GitHub] [hudi] NickYoungPeng commented on pull request #2744: [HUDI-1742]improve table level config priority

2021-04-06 Thread GitBox
NickYoungPeng commented on pull request #2744: URL: https://github.com/apache/hudi/pull/2744#issuecomment-814594032 > @NickYoungPeng Please refer these 2 classes - `TestHoodieMultiTableDeltaStreamer` and `TestHoodieDeltaStreamer`. Hopefully these will give you a decent idea about writing

[GitHub] [hudi] wangxianghu edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition

2021-04-06 Thread GitBox
wangxianghu edited a comment on pull request #2452: URL: https://github.com/apache/hudi/pull/2452#issuecomment-814591829 > I understand its in darft. Just before you open it up for reviews, can you add more info in the description. > is this a standalone tool? or to be used as a

[GitHub] [hudi] wangxianghu commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition

2021-04-06 Thread GitBox
wangxianghu commented on pull request #2452: URL: https://github.com/apache/hudi/pull/2452#issuecomment-814591829 > I understand its in darft. Just before you open it up for reviews, can you add more info in the description. > is this a standalone tool? or to be used as a strategy along

[GitHub] [hudi] nsivabalan commented on a change in pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-06 Thread GitBox
nsivabalan commented on a change in pull request #2765: URL: https://github.com/apache/hudi/pull/2765#discussion_r608313291 ## File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala ## @@ -49,7 +51,48 @@ object AvroConversionUtils {

[GitHub] [hudi] codecov-io commented on pull request #2779: [HUDI-1767]Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink

2021-04-06 Thread GitBox
codecov-io commented on pull request #2779: URL: https://github.com/apache/hudi/pull/2779#issuecomment-814573470 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2779?src=pr=h1) Report > Merging [#2779](https://codecov.io/gh/apache/hudi/pull/2779?src=pr=desc) (d9d3bd0) into

[GitHub] [hudi] nsivabalan commented on issue #2774: [SUPPORT] NoSuchMethodError on exposing metrics

2021-04-06 Thread GitBox
nsivabalan commented on issue #2774: URL: https://github.com/apache/hudi/issues/2774#issuecomment-814567349 @n3nash : can you help here or loop in someone who can take a look at this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hudi] nsivabalan commented on issue #2771: [SUPPORT] Log files are not compacted

2021-04-06 Thread GitBox
nsivabalan commented on issue #2771: URL: https://github.com/apache/hudi/issues/2771#issuecomment-814566135 I see you have enabled async compaction. If you enable inline compaction, does it work? -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] nsivabalan commented on issue #2770: [SUPPORT] How column _hoodie_is_deleted works?

2021-04-06 Thread GitBox
nsivabalan commented on issue #2770: URL: https://github.com/apache/hudi/issues/2770#issuecomment-814563717 @rubenssoto : sorry what happened. why closed the issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [hudi] nsivabalan commented on pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-06 Thread GitBox
nsivabalan commented on pull request #2765: URL: https://github.com/apache/hudi/pull/2765#issuecomment-814563007 @lw309637554 : may be you wanna review this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] nsivabalan commented on a change in pull request #2776: [HUDI-1768] spark datasource support schema validate add column

2021-04-06 Thread GitBox
nsivabalan commented on a change in pull request #2776: URL: https://github.com/apache/hudi/pull/2776#discussion_r608304445 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala ## @@ -669,4 +669,90 @@ class

[GitHub] [hudi] nsivabalan commented on a change in pull request #2776: [HUDI-1768] spark datasource support schema validate add column

2021-04-06 Thread GitBox
nsivabalan commented on a change in pull request #2776: URL: https://github.com/apache/hudi/pull/2776#discussion_r608304045 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala ## @@ -669,4 +669,90 @@ class

[GitHub] [hudi] nsivabalan commented on a change in pull request #2776: [HUDI-1768] spark datasource support schema validate add column

2021-04-06 Thread GitBox
nsivabalan commented on a change in pull request #2776: URL: https://github.com/apache/hudi/pull/2776#discussion_r608303595 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala ## @@ -669,4 +669,90 @@ class

[GitHub] [hudi] nsivabalan commented on a change in pull request #2776: [HUDI-1768] spark datasource support schema validate add column

2021-04-06 Thread GitBox
nsivabalan commented on a change in pull request #2776: URL: https://github.com/apache/hudi/pull/2776#discussion_r608303303 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala ## @@ -669,4 +669,90 @@ class

[jira] [Updated] (HUDI-1767) Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink

2021-04-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1767: - Labels: pull-request-available (was: ) > Add setter to HoodieKey and HoodieRecordLocation to

[GitHub] [hudi] MyLanPangzi opened a new pull request #2779: [HUDI-1767]Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink

2021-04-06 Thread GitBox
MyLanPangzi opened a new pull request #2779: URL: https://github.com/apache/hudi/pull/2779 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of

[GitHub] [hudi] nsivabalan commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-04-06 Thread GitBox
nsivabalan commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-814557848 @satishkotha : I haven't gone through entire comment history. but we have introduced new labels to assist us in better help community. We have these labels for issues.

[GitHub] [hudi] nsivabalan merged pull request #2753: [HUDI-1750] Fail to load user's class if user move hudi-spark-bundle jar into spark classpath

2021-04-06 Thread GitBox
nsivabalan merged pull request #2753: URL: https://github.com/apache/hudi/pull/2753 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[hudi] branch master updated: [HUDI-1750] Fail to load user's class if user move hudi-spark-bundle jar into spark classpath (#2753)

2021-04-06 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 8527590 [HUDI-1750] Fail to load user's class

[GitHub] [hudi] nsivabalan commented on pull request #2753: [HUDI-1750] Fail to load user's class if user move hudi-spark-bundle jar into spark classpath

2021-04-06 Thread GitBox
nsivabalan commented on pull request #2753: URL: https://github.com/apache/hudi/pull/2753#issuecomment-814556608 sounds good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] nsivabalan edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition

2021-04-06 Thread GitBox
nsivabalan edited a comment on pull request #2452: URL: https://github.com/apache/hudi/pull/2452#issuecomment-814556277 I understand its in darft. Just before you open it up for reviews, can you add more info in the description. is this a standalone tool? or to be used as a strategy

[GitHub] [hudi] nsivabalan commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition

2021-04-06 Thread GitBox
nsivabalan commented on pull request #2452: URL: https://github.com/apache/hudi/pull/2452#issuecomment-814556277 Can you add more info in the description. is this a standalone tool? or to be used as a strategy along w/ cleaning? or hudi-cli ? -- This is an automated message from

[GitHub] [hudi] codecov-io edited a comment on pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-04-06 Thread GitBox
codecov-io edited a comment on pull request #2325: URL: https://github.com/apache/hudi/pull/2325#issuecomment-742860619 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2325?src=pr=h1) Report > Merging [#2325](https://codecov.io/gh/apache/hudi/pull/2325?src=pr=desc) (d5facf4) into

[GitHub] [hudi] codecov-io edited a comment on pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-04-06 Thread GitBox
codecov-io edited a comment on pull request #2325: URL: https://github.com/apache/hudi/pull/2325#issuecomment-742860619 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2325?src=pr=h1) Report > Merging [#2325](https://codecov.io/gh/apache/hudi/pull/2325?src=pr=desc) (d5facf4) into

[GitHub] [hudi] codecov-io edited a comment on pull request #2744: [HUDI-1742]improve table level config priority

2021-04-06 Thread GitBox
codecov-io edited a comment on pull request #2744: URL: https://github.com/apache/hudi/pull/2744#issuecomment-810101277 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2744?src=pr=h1) Report > Merging [#2744](https://codecov.io/gh/apache/hudi/pull/2744?src=pr=desc) (ed1033d) into

[GitHub] [hudi] codecov-io edited a comment on pull request #1946: [HUDI-1176]Upgrade tp log4j2

2021-04-06 Thread GitBox
codecov-io edited a comment on pull request #1946: URL: https://github.com/apache/hudi/pull/1946#issuecomment-774846457 # [Codecov](https://codecov.io/gh/apache/hudi/pull/1946?src=pr=h1) Report > Merging [#1946](https://codecov.io/gh/apache/hudi/pull/1946?src=pr=desc) (77d3697) into

[GitHub] [hudi] codecov-io edited a comment on pull request #1946: [HUDI-1176]Upgrade tp log4j2

2021-04-06 Thread GitBox
codecov-io edited a comment on pull request #1946: URL: https://github.com/apache/hudi/pull/1946#issuecomment-774846457 # [Codecov](https://codecov.io/gh/apache/hudi/pull/1946?src=pr=h1) Report > Merging [#1946](https://codecov.io/gh/apache/hudi/pull/1946?src=pr=desc) (77d3697) into

[GitHub] [hudi] codecov-io edited a comment on pull request #1946: [HUDI-1176]Upgrade tp log4j2

2021-04-06 Thread GitBox
codecov-io edited a comment on pull request #1946: URL: https://github.com/apache/hudi/pull/1946#issuecomment-774846457 # [Codecov](https://codecov.io/gh/apache/hudi/pull/1946?src=pr=h1) Report > Merging [#1946](https://codecov.io/gh/apache/hudi/pull/1946?src=pr=desc) (77d3697) into

[GitHub] [hudi] xiarixiaoyao edited a comment on pull request #2720: [HUDI-1719]hive on spark/mr,Incremental query of the mor table, the partition field is incorrect

2021-04-06 Thread GitBox
xiarixiaoyao edited a comment on pull request #2720: URL: https://github.com/apache/hudi/pull/2720#issuecomment-814535223 sorry for later。 i will add UT, thanks @nsivabalan @garyli1019 another question: why test in TestHoodieCombineHiveInputFormat is Disabled ? @Test

[jira] [Created] (HUDI-1771) Keep the change flags from CDC source for Flink writer

2021-04-06 Thread Danny Chen (Jira)
Danny Chen created HUDI-1771: Summary: Keep the change flags from CDC source for Flink writer Key: HUDI-1771 URL: https://issues.apache.org/jira/browse/HUDI-1771 Project: Apache Hudi Issue Type:

[GitHub] [hudi] zherenyu831 commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-04-06 Thread GitBox
zherenyu831 commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-814540364 @satishkotha Thank you for the explaining. Totally understood. Since insert_overwrite_table every time creating version of files, in this case, instead of using

[GitHub] [hudi] xiarixiaoyao commented on pull request #2720: [HUDI-1719]hive on spark/mr,Incremental query of the mor table, the partition field is incorrect

2021-04-06 Thread GitBox
xiarixiaoyao commented on pull request #2720: URL: https://github.com/apache/hudi/pull/2720#issuecomment-814535223 sorry for later。 i will add UT, thanks @nsivabalan @garyli1019 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] kimberlyamandalu commented on issue #2696: Metadata and runtime exceptions in Hudi 0.7.0 on AWS Glue

2021-04-06 Thread GitBox
kimberlyamandalu commented on issue #2696: URL: https://github.com/apache/hudi/issues/2696#issuecomment-814531773 Might be related to this? https://stackoverflow.com/questions/39498492/spark-and-amazon-emr-s3-connections-not-being-closed -- This is an automated message from the Apache

[GitHub] [hudi] kimberlyamandalu edited a comment on issue #2696: Metadata and runtime exceptions in Hudi 0.7.0 on AWS Glue

2021-04-06 Thread GitBox
kimberlyamandalu edited a comment on issue #2696: URL: https://github.com/apache/hudi/issues/2696#issuecomment-814530289 Hello @n3nash I was able to get a little more info. Pls see log output below. Looks like the exception is being thrown by the AWS SDK. But not sure what it

[GitHub] [hudi] kimberlyamandalu edited a comment on issue #2696: Metadata and runtime exceptions in Hudi 0.7.0 on AWS Glue

2021-04-06 Thread GitBox
kimberlyamandalu edited a comment on issue #2696: URL: https://github.com/apache/hudi/issues/2696#issuecomment-814530289 Hello @n3nash I was able to get a little more info. Pls see log output below. Looks like the exception is being thrown by the AWS SDK. But not sure what it

[GitHub] [hudi] kimberlyamandalu commented on issue #2696: Metadata and runtime exceptions in Hudi 0.7.0 on AWS Glue

2021-04-06 Thread GitBox
kimberlyamandalu commented on issue #2696: URL: https://github.com/apache/hudi/issues/2696#issuecomment-814530289 Hello @n3nash I was able to get a little more info. Pls see log output below. Looks like the exception is being thrown by the AWS SDK. But not sure what it means.

[hudi] branch master updated: [MINOR] Fix deprecated build link for travis (#2778)

2021-04-06 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new e692c70 [MINOR] Fix deprecated build link for

[GitHub] [hudi] yanghua merged pull request #2778: [MINOR] Fix deprecated build link for travis

2021-04-06 Thread GitBox
yanghua merged pull request #2778: URL: https://github.com/apache/hudi/pull/2778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] wangxianghu commented on pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-04-06 Thread GitBox
wangxianghu commented on pull request #2325: URL: https://github.com/apache/hudi/pull/2325#issuecomment-814524534 @hddong Thanks for addressing my concerns. I left one comment you can consider, after that, we can land this PR. cc @yanghua -- This is an automated message from the

[GitHub] [hudi] wangxianghu commented on a change in pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-04-06 Thread GitBox
wangxianghu commented on a change in pull request #2325: URL: https://github.com/apache/hudi/pull/2325#discussion_r608268346 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java ## @@ -84,6 +88,14 @@ public

[GitHub] [hudi] bvaradar commented on issue #2294: [SUPPORT] java.lang.IllegalArgumentException: Can not create a Path from an empty string on non partitioned COW table

2021-04-06 Thread GitBox
bvaradar commented on issue #2294: URL: https://github.com/apache/hudi/issues/2294#issuecomment-814500673 @vansimonsen : Can you open a new GH issue with the stack trace . @rubenssoto : I believe the PR landed before 0.7.0 was cut. -- This is an automated message from the Apache Git

[GitHub] [hudi] hmit opened a new pull request #2778: Fix deprecated build link

2021-04-06 Thread GitBox
hmit opened a new pull request #2778: URL: https://github.com/apache/hudi/pull/2778 ## What is the purpose of the pull request The travisci link to build status has moved, updating to fetch from .com vs .org ## Committer checklist - [ ] Commit message is descriptive of the

[jira] [Updated] (HUDI-1770) Deltastreamer throws errors when not running frequently

2021-04-06 Thread Vinoth Govindarajan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Govindarajan updated HUDI-1770: -- Affects Version/s: 0.7.0 > Deltastreamer throws errors when not running frequently >

[jira] [Updated] (HUDI-1770) Deltastreamer throws errors when not running frequently

2021-04-06 Thread Vinoth Govindarajan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Govindarajan updated HUDI-1770: -- Affects Version/s: 0.8.0 > Deltastreamer throws errors when not running frequently >

[GitHub] [hudi] vansimonsen commented on issue #2294: [SUPPORT] java.lang.IllegalArgumentException: Can not create a Path from an empty string on non partitioned COW table

2021-04-06 Thread GitBox
vansimonsen commented on issue #2294: URL: https://github.com/apache/hudi/issues/2294#issuecomment-814439644 > @bvaradar is it expected that this bug exists on 0.7.0? > What problem this bug cause? I'm having same issue on new `0.8.0` -- This is an automated message from the

[jira] [Updated] (HUDI-1770) Deltastreamer throws errors when not running frequently

2021-04-06 Thread Vinoth Govindarajan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Govindarajan updated HUDI-1770: -- Description: When delta streamer is using HoodieIncrSource from another parent Hudi

[jira] [Created] (HUDI-1770) Deltastreamer throws errors when not running frequently

2021-04-06 Thread Vinoth Govindarajan (Jira)
Vinoth Govindarajan created HUDI-1770: - Summary: Deltastreamer throws errors when not running frequently Key: HUDI-1770 URL: https://issues.apache.org/jira/browse/HUDI-1770 Project: Apache Hudi

svn commit: r46896 - in /release/hudi/0.8.0: ./ hudi-0.8.0.src.tgz hudi-0.8.0.src.tgz.asc hudi-0.8.0.src.tgz.sha512

2021-04-06 Thread sivabalan
Author: sivabalan Date: Tue Apr 6 20:02:34 2021 New Revision: 46896 Log: Adding release artifacts for release-0.8.0 Added: release/hudi/0.8.0/ release/hudi/0.8.0/hudi-0.8.0.src.tgz (with props) release/hudi/0.8.0/hudi-0.8.0.src.tgz.asc

[GitHub] [hudi] codecov-io commented on pull request #2777: [HUDI-1615] Fixing usage of NULL schema for delete operation in HoodieSparkSqlWriter

2021-04-06 Thread GitBox
codecov-io commented on pull request #2777: URL: https://github.com/apache/hudi/pull/2777#issuecomment-814337083 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2777?src=pr=h1) Report > Merging [#2777](https://codecov.io/gh/apache/hudi/pull/2777?src=pr=desc) (d643402) into

[GitHub] [hudi] satishkotha commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-04-06 Thread GitBox
satishkotha commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-814291398 Hi, Why do you think it breaks isolation? We create new version of data files with new fileIds and clean up old versions. With KEEP_LATEST_VERSIONS cleaner policy, we retain

[GitHub] [hudi] nsivabalan commented on pull request #2655: [WIP] [HUDI-1615] Fixing null schema for delete operation in spark datasource

2021-04-06 Thread GitBox
nsivabalan commented on pull request #2655: URL: https://github.com/apache/hudi/pull/2655#issuecomment-814290764 Closing this in favor of https://github.com/apache/hudi/pull/2777 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] nsivabalan closed pull request #2655: [WIP] [HUDI-1615] Fixing null schema for delete operation in spark datasource

2021-04-06 Thread GitBox
nsivabalan closed pull request #2655: URL: https://github.com/apache/hudi/pull/2655 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [hudi] nsivabalan opened a new pull request #2777: [HUDI-1615] Fixing usage of NULL schema for delete operation in HoodieSparkSqlWriter

2021-04-06 Thread GitBox
nsivabalan opened a new pull request #2777: URL: https://github.com/apache/hudi/pull/2777 ## What is the purpose of the pull request *Fixing usage of NULL schema for delete operation in HoodieSparkSqlWriter* ## Brief change log - *Fixing usage of NULL schema for

[jira] [Commented] (HUDI-1758) Flink insert command does not update the record

2021-04-06 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17315726#comment-17315726 ] Nishith Agarwal commented on HUDI-1758: --- [~danny0405] Re-added the screenshot. I will check the logs

[jira] [Updated] (HUDI-1758) Flink insert command does not update the record

2021-04-06 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1758: -- Attachment: Screen Shot 2021-04-02 at 12.10.08 AM.png > Flink insert command does not update

[jira] [Updated] (HUDI-1758) Flink insert command does not update the record

2021-04-06 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1758: -- Description:    !Screen Shot 2021-04-02 at 12.10.08 AM.png! Followed the steps mentioned in

[jira] [Updated] (HUDI-1758) Flink insert command does not update the record

2021-04-06 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishith Agarwal updated HUDI-1758: -- Description:   !Screen Shot 2021-04-02 at 12.10.08 AM.png! Followed the steps mentioned in

[jira] [Assigned] (HUDI-1769) websites updates for 0.8.0 release

2021-04-06 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-1769: - Assignee: Gary Li > websites updates for 0.8.0 release >

[jira] [Created] (HUDI-1769) websites updates for 0.8.0 release

2021-04-06 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-1769: - Summary: websites updates for 0.8.0 release Key: HUDI-1769 URL: https://issues.apache.org/jira/browse/HUDI-1769 Project: Apache Hudi Issue Type:

[GitHub] [hudi] codecov-io commented on pull request #2776: [HUDI-1768] spark datasource support schema validate add column

2021-04-06 Thread GitBox
codecov-io commented on pull request #2776: URL: https://github.com/apache/hudi/pull/2776#issuecomment-814206544 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2776?src=pr=h1) Report > Merging [#2776](https://codecov.io/gh/apache/hudi/pull/2776?src=pr=desc) (4c3ff43) into

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support

2021-04-06 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2645: URL: https://github.com/apache/hudi/pull/2645#discussion_r607469389 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/MergeIntoTest.scala ## @@ -0,0 +1,183 @@ +/* + * Licensed to

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support

2021-04-06 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2645: URL: https://github.com/apache/hudi/pull/2645#discussion_r607943535 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/MergeIntoTest.scala ## @@ -0,0 +1,183 @@ +/* + * Licensed to

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support

2021-04-06 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2645: URL: https://github.com/apache/hudi/pull/2645#discussion_r607463859 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/MergeIntoTest.scala ## @@ -0,0 +1,183 @@ +/* + * Licensed to

[jira] [Updated] (HUDI-1768) spark datasource support schema validate add column

2021-04-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1768: - Labels: pull-request-available (was: ) > spark datasource support schema validate add column >

[GitHub] [hudi] lw309637554 opened a new pull request #2776: [HUDI-1768] spark datasource support schema validate add column

2021-04-06 Thread GitBox
lw309637554 opened a new pull request #2776: URL: https://github.com/apache/hudi/pull/2776 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of

[GitHub] [hudi] rubenssoto closed issue #2770: [SUPPORT] How column _hoodie_is_deleted works?

2021-04-06 Thread GitBox
rubenssoto closed issue #2770: URL: https://github.com/apache/hudi/issues/2770 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] zherenyu831 edited a comment on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-04-06 Thread GitBox
zherenyu831 edited a comment on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-814157136 @satishkotha cc @bvaradar @vinothchandar I found the problem why cleaner only keep one file version

[GitHub] [hudi] zherenyu831 commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-04-06 Thread GitBox
zherenyu831 commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-814157136 @satishkotha cc @bvaradar I found the problem why cleaner only keep one file version

[GitHub] [hudi] NickYoungPeng removed a comment on pull request #2744: [HUDI-1742]improve table level config priority

2021-04-06 Thread GitBox
NickYoungPeng removed a comment on pull request #2744: URL: https://github.com/apache/hudi/pull/2744#issuecomment-810062733 @yanghua -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[jira] [Updated] (HUDI-1768) spark datasource support schema validate add column

2021-04-06 Thread liwei (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liwei updated HUDI-1768: Issue Type: Improvement (was: Bug) > spark datasource support schema validate add column >

[jira] [Assigned] (HUDI-1768) spark datasource support schema validate add column

2021-04-06 Thread liwei (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liwei reassigned HUDI-1768: --- Assignee: liwei > spark datasource support schema validate add column >

[jira] [Created] (HUDI-1768) spark datasource support schema validate add column

2021-04-06 Thread liwei (Jira)
liwei created HUDI-1768: --- Summary: spark datasource support schema validate add column Key: HUDI-1768 URL: https://issues.apache.org/jira/browse/HUDI-1768 Project: Apache Hudi Issue Type: Bug

[GitHub] [hudi] nsivabalan commented on issue #2743: Do we have any TTL mechanism in Hudi?

2021-04-06 Thread GitBox
nsivabalan commented on issue #2743: URL: https://github.com/apache/hudi/issues/2743#issuecomment-814143042 @n3nash @vinothchandar @bvaradar : any thoughts here or workarounds until we have per record support for TTL. -- This is an automated message from the Apache Git Service. To

[jira] [Updated] (HUDI-1740) insert_overwrite_table and insert_overwrite first replacecommit has empty partitionToReplaceFileIds

2021-04-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1740: - Labels: pull-request-available (was: ) > insert_overwrite_table and insert_overwrite first

[GitHub] [hudi] ssdong closed pull request #2775: [HUDI-1740] fix empty partitionToReplaceFileIds

2021-04-06 Thread GitBox
ssdong closed pull request #2775: URL: https://github.com/apache/hudi/pull/2775 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] ssdong opened a new pull request #2775: [HUDI-1740] fix empty partitionToReplaceFileIds

2021-04-06 Thread GitBox
ssdong opened a new pull request #2775: URL: https://github.com/apache/hudi/pull/2775 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the

[jira] [Commented] (HUDI-1657) build failed on AArch64, Fedora 33

2021-04-06 Thread Lutz Weischer (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17315502#comment-17315502 ] Lutz Weischer commented on HUDI-1657: - Tried fresh again, using Java 1.8, 11 and 16, but it fails. >

[GitHub] [hudi] Karl-WangSK closed pull request #2106: [HUDI-1284] preCombine all HoodieRecords and update all fields according to orderingVal

2021-04-06 Thread GitBox
Karl-WangSK closed pull request #2106: URL: https://github.com/apache/hudi/pull/2106 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[jira] [Closed] (HUDI-1757) Assigns the buckets by record key for Flink writer

2021-04-06 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1757. -- Resolution: Done 9c369c607df2816ea2cd1221fb6d879e3fb8f74c > Assigns the buckets by record key for Flink writer

[jira] [Updated] (HUDI-1757) Assigns the buckets by record key for Flink writer

2021-04-06 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-1757: --- Fix Version/s: 0.9.0 > Assigns the buckets by record key for Flink writer >

[hudi] branch master updated: [HUDI-1757] Assigns the buckets by record key for Flink writer (#2757)

2021-04-06 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 9c369c6 [HUDI-1757] Assigns the buckets by

[GitHub] [hudi] yanghua merged pull request #2757: [HUDI-1757] Assigns the buckets by record key for Flink writer

2021-04-06 Thread GitBox
yanghua merged pull request #2757: URL: https://github.com/apache/hudi/pull/2757 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] ssdong commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-04-06 Thread GitBox
ssdong commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-813959142 The original issue being brought up by @zherenyu831 has a bigger impact than we had imagined. The simple fix to omit the empty `partitionToReplaceFileIds` would work on release `0.7.0`

[jira] [Commented] (HUDI-1767) Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink

2021-04-06 Thread Jira
[ https://issues.apache.org/jira/browse/HUDI-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17315364#comment-17315364 ] 谢波 commented on HUDI-1767: -- Can you assign to me? [~danny0405] > Add setter to HoodieKey and

[jira] [Commented] (HUDI-1740) insert_overwrite_table and insert_overwrite first replacecommit has empty partitionToReplaceFileIds

2021-04-06 Thread Susu Dong (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17315361#comment-17315361 ] Susu Dong commented on HUDI-1740: - [~jagmeet.bali]  [~satish] The fix for checking whether

[jira] [Updated] (HUDI-1767) Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink

2021-04-06 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-1767: - Description: Currently the {{HoodieKey}} and {{HoodieRecordLocation}} do not implement the setter for the

[jira] [Updated] (HUDI-1767) Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink

2021-04-06 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-1767: - Description: Currently the {{HoodieKey}} and {{HoodieRecordLocation}} do not implement the setters for

[jira] [Created] (HUDI-1767) Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink

2021-04-06 Thread Danny Chen (Jira)
Danny Chen created HUDI-1767: Summary: Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink Key: HUDI-1767 URL: https://issues.apache.org/jira/browse/HUDI-1767

[GitHub] [hudi] danny0405 commented on a change in pull request #2757: [HUDI-1757] Assigns the buckets by record key for Flink writer

2021-04-06 Thread GitBox
danny0405 commented on a change in pull request #2757: URL: https://github.com/apache/hudi/pull/2757#discussion_r607659383 ## File path: hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java ## @@ -78,19 +71,19 @@ private final Configuration

[GitHub] [hudi] yanghua commented on a change in pull request #2757: [HUDI-1757] Assigns the buckets by record key for Flink writer

2021-04-06 Thread GitBox
yanghua commented on a change in pull request #2757: URL: https://github.com/apache/hudi/pull/2757#discussion_r607642888 ## File path: hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java ## @@ -78,19 +71,19 @@ private final Configuration conf;

[jira] [Assigned] (HUDI-1740) insert_overwrite_table and insert_overwrite first replacecommit has empty partitionToReplaceFileIds

2021-04-06 Thread Susu Dong (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Susu Dong reassigned HUDI-1740: --- Assignee: Susu Dong > insert_overwrite_table and insert_overwrite first replacecommit has empty >

[GitHub] [hudi] codecov-io edited a comment on pull request #2754: [HUDI-1751] Remove irrelevant properties from passing to kafkaConsumer which in turn prints lot of warn logs

2021-04-06 Thread GitBox
codecov-io edited a comment on pull request #2754: URL: https://github.com/apache/hudi/pull/2754#issuecomment-811625350 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2754?src=pr=h1) Report > Merging [#2754](https://codecov.io/gh/apache/hudi/pull/2754?src=pr=desc) (b61ec85) into

[GitHub] [hudi] ssdong edited a comment on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-04-06 Thread GitBox
ssdong edited a comment on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-813825461 @nsivabalan @bvaradar Sorry for the bother. Could you kindly add me as a contributor so I could self-assign issues? I've set up my local dev env successfully and would like to

[jira] [Updated] (HUDI-1764) Add support for Hudi CLI tools to schedule and run clustering

2021-04-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1764: - Labels: pull-request-available (was: ) > Add support for Hudi CLI tools to schedule and run

[GitHub] [hudi] codecov-io commented on pull request #2773: [HUDI-1764] Cli clustering support

2021-04-06 Thread GitBox
codecov-io commented on pull request #2773: URL: https://github.com/apache/hudi/pull/2773#issuecomment-813928206 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2773?src=pr=h1) Report > Merging [#2773](https://codecov.io/gh/apache/hudi/pull/2773?src=pr=desc) (a85a0fe) into

[jira] [Comment Edited] (HUDI-1657) build failed on AArch64, Fedora 33

2021-04-06 Thread shenjinxin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313672#comment-17313672 ] shenjinxin edited comment on HUDI-1657 at 4/6/21, 7:56 AM: --- I also encounter the

[GitHub] [hudi] vobukh opened a new issue #2774: [SUPPORT] NoSuchMethodError on exposing metrics

2021-04-06 Thread GitBox
vobukh opened a new issue #2774: URL: https://github.com/apache/hudi/issues/2774 I'm trying to expose hudi metrics to prometheus push gateway. During the write operation I've got an error: > java.lang.NoSuchMethodError:

[GitHub] [hudi] pratyakshsharma commented on pull request #2744: [HUDI-1742]improve table level config priority

2021-04-06 Thread GitBox
pratyakshsharma commented on pull request #2744: URL: https://github.com/apache/hudi/pull/2744#issuecomment-813898486 @NickYoungPeng Please refer these 2 classes - `TestHoodieMultiTableDeltaStreamer` and `TestHoodieDeltaStreamer`. Hopefully these will give you a decent idea about writing

  1   2   >