[GitHub] [hudi] hddong commented on pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-04-07 Thread GitBox
hddong commented on pull request #2325: URL: https://github.com/apache/hudi/pull/2325#issuecomment-814734415 @wangxianghu: It's OK now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hudi] codecov-io edited a comment on pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
codecov-io edited a comment on pull request #2773: URL: https://github.com/apache/hudi/pull/2773#issuecomment-813928206 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2773?src=pr=h1) Report > Merging [#2773](https://codecov.io/gh/apache/hudi/pull/2773?src=pr=desc) (fc2f340) into

[GitHub] [hudi] simon824 opened a new pull request #2782: [MINOR] Optimized code

2021-04-07 Thread GitBox
simon824 opened a new pull request #2782: URL: https://github.com/apache/hudi/pull/2782 ## What is the purpose of the pull request Optimized some code ## Brief change log ## Verify this pull request ## Committer checklist - [ ] Has a corresponding

[jira] [Updated] (HUDI-1551) Support Partition with BigDecimal/Integer field

2021-04-07 Thread Chanh Le (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chanh Le updated HUDI-1551: --- Summary: Support Partition with BigDecimal/Integer field (was: Support Partition with BigDecimal field) >

[GitHub] [hudi] yanghua merged pull request #2779: [HUDI-1767] Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink

2021-04-07 Thread GitBox
yanghua merged pull request #2779: URL: https://github.com/apache/hudi/pull/2779 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[hudi] branch master updated: [HUDI-1767] Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink (#2779)

2021-04-07 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new d035fcb [HUDI-1767] Add setter to HoodieKey

[GitHub] [hudi] lw309637554 commented on pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-07 Thread GitBox
lw309637554 commented on pull request #2765: URL: https://github.com/apache/hudi/pull/2765#issuecomment-814636227 > @lw309637554 : may be you wanna review this. @nsivabalan thanks , if this pr land, https://github.com/apache/hudi/pull/2776 will be ok , and i add more test for

[jira] [Assigned] (HUDI-1773) HoodieFileGroup code optimize

2021-04-07 Thread Jira
[ https://issues.apache.org/jira/browse/HUDI-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 谢波 reassigned HUDI-1773: Assignee: 谢波 > HoodieFileGroup code optimize > - > > Key: HUDI-1773 >

[jira] [Updated] (HUDI-1772) HoodieFileGroupId compareTo logical error(fileId self compare)

2021-04-07 Thread Jira
[ https://issues.apache.org/jira/browse/HUDI-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 谢波 updated HUDI-1772: - Attachment: (was: image-2021-04-07-14-23-25-271.png) > HoodieFileGroupId compareTo logical error(fileId self compare)

[jira] [Created] (HUDI-1773) HoodieFileGroup code optimize

2021-04-07 Thread Jira
谢波 created HUDI-1773: Summary: HoodieFileGroup code optimize Key: HUDI-1773 URL: https://issues.apache.org/jira/browse/HUDI-1773 Project: Apache Hudi Issue Type: Improvement Components: Common

[GitHub] [hudi] MyLanPangzi opened a new pull request #2781: [HUDI-1773]HoodieFileGroup code optimize

2021-04-07 Thread GitBox
MyLanPangzi opened a new pull request #2781: URL: https://github.com/apache/hudi/pull/2781 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose

[jira] [Updated] (HUDI-1773) HoodieFileGroup code optimize

2021-04-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1773: - Labels: pull-request-available (was: ) > HoodieFileGroup code optimize >

[GitHub] [hudi] codecov-io edited a comment on pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support

2021-04-07 Thread GitBox
codecov-io edited a comment on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-792430670 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=h1) Report > Merging [#2645](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=desc) (3e93ac6) into

[GitHub] [hudi] lw309637554 commented on a change in pull request #2773: [HUDI-1764] Hudi-CLI support for clustering

2021-04-07 Thread GitBox
lw309637554 commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r608399250 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieClusteringJob.java ## @@ -156,15 +155,15 @@ private int

[GitHub] [hudi] lw309637554 commented on a change in pull request #2773: [HUDI-1764] Hudi-CLI support for clustering

2021-04-07 Thread GitBox
lw309637554 commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r608401545 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/SparkMain.java ## @@ -159,6 +161,31 @@ public static void main(String[] args) throws

[GitHub] [hudi] codecov-io commented on pull request #2780: [HUDI-1772] HoodieFileGroupId compareTo logical error(fileId self compare)

2021-04-07 Thread GitBox
codecov-io commented on pull request #2780: URL: https://github.com/apache/hudi/pull/2780#issuecomment-814672175 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2780?src=pr=h1) Report > Merging [#2780](https://codecov.io/gh/apache/hudi/pull/2780?src=pr=desc) (8257ade) into

[GitHub] [hudi] n3nash commented on issue #2774: [SUPPORT] NoSuchMethodError on exposing metrics

2021-04-07 Thread GitBox
n3nash commented on issue #2774: URL: https://github.com/apache/hudi/issues/2774#issuecomment-814678338 @vobukh This looks like a problem due to the version differences between `io.dropwizard` in [prometheus-java

[GitHub] [hudi] jintaoguan commented on a change in pull request #2773: [HUDI-1764] Hudi-CLI support for clustering

2021-04-07 Thread GitBox
jintaoguan commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r608409284 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/SparkMain.java ## @@ -159,6 +161,31 @@ public static void main(String[] args) throws

[GitHub] [hudi] n3nash commented on issue #2771: [SUPPORT] Log files are not compacted

2021-04-07 Thread GitBox
n3nash commented on issue #2771: URL: https://github.com/apache/hudi/issues/2771#issuecomment-814682374 @stackfun I see that you have some conflicting configs ``` "hoodie.datasource.compaction.async.enable": True, "hoodie.index.type": "SIMPLE",

[GitHub] [hudi] jintaoguan commented on a change in pull request #2773: [HUDI-1764] Hudi-CLI support for clustering

2021-04-07 Thread GitBox
jintaoguan commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r608409284 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/SparkMain.java ## @@ -159,6 +161,31 @@ public static void main(String[] args) throws

[GitHub] [hudi] n3nash commented on issue #2696: Metadata and runtime exceptions in Hudi 0.7.0 on AWS Glue

2021-04-07 Thread GitBox
n3nash commented on issue #2696: URL: https://github.com/apache/hudi/issues/2696#issuecomment-814684654 @kimberlyamandalu This is unknown territory for me as well. Let me loop in some AWS experts here. @umehrot2 Do you have any idea what the timeouts may be related to ? To summarize this

[hudi] branch master updated: [HUDI-1751] DeltaStreamer print many unnecessary warn log (#2754)

2021-04-07 Thread nagarwal
This is an automated email from the ASF dual-hosted git repository. nagarwal pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new dadd081 [HUDI-1751] DeltaStreamer print many

[GitHub] [hudi] n3nash merged pull request #2754: [HUDI-1751] Remove irrelevant properties from passing to kafkaConsumer which in turn prints lot of warn logs

2021-04-07 Thread GitBox
n3nash merged pull request #2754: URL: https://github.com/apache/hudi/pull/2754 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] jintaoguan commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
jintaoguan commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r608427661 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieClusteringJob.java ## @@ -156,15 +155,15 @@ private int

[GitHub] [hudi] jintaoguan commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
jintaoguan commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r608427661 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieClusteringJob.java ## @@ -156,15 +155,15 @@ private int

[GitHub] [hudi] aditiwari01 commented on a change in pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-07 Thread GitBox
aditiwari01 commented on a change in pull request #2765: URL: https://github.com/apache/hudi/pull/2765#discussion_r608433997 ## File path: hudi-utilities/src/test/resources/delta-streamer-config/source-jdbc.avsc ## @@ -26,34 +26,42 @@ }, { "name": "TIMESTAMP", -

[GitHub] [hudi] aditiwari01 commented on a change in pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-07 Thread GitBox
aditiwari01 commented on a change in pull request #2765: URL: https://github.com/apache/hudi/pull/2765#discussion_r608434413 ## File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestAvroConversionUtils.scala ## @@ -0,0 +1,85 @@ +/* + * Licensed to the

[GitHub] [hudi] codecov-io edited a comment on pull request #2283: [HUDI-1415] Read Hoodie Table As Spark DataSource Table

2021-04-07 Thread GitBox
codecov-io edited a comment on pull request #2283: URL: https://github.com/apache/hudi/pull/2283#issuecomment-734137301 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2283?src=pr=h1) Report > Merging [#2283](https://codecov.io/gh/apache/hudi/pull/2283?src=pr=desc) (ba70819) into

[jira] [Updated] (HUDI-1772) HoodieFileGroupId compareTo logical error(fileId self compare)

2021-04-07 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1772: - Labels: pull-request-available (was: ) > HoodieFileGroupId compareTo logical error(fileId self

[GitHub] [hudi] codecov-io edited a comment on pull request #2283: [HUDI-1415] Read Hoodie Table As Spark DataSource Table

2021-04-07 Thread GitBox
codecov-io edited a comment on pull request #2283: URL: https://github.com/apache/hudi/pull/2283#issuecomment-734137301 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2283?src=pr=h1) Report > Merging [#2283](https://codecov.io/gh/apache/hudi/pull/2283?src=pr=desc) (ba70819) into

[GitHub] [hudi] jintaoguan commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
jintaoguan commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r608409284 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/SparkMain.java ## @@ -159,6 +161,31 @@ public static void main(String[] args) throws

[GitHub] [hudi] aditiwari01 commented on a change in pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-07 Thread GitBox
aditiwari01 commented on a change in pull request #2765: URL: https://github.com/apache/hudi/pull/2765#discussion_r608436934 ## File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala ## @@ -49,7 +51,48 @@ object AvroConversionUtils {

[GitHub] [hudi] lw309637554 commented on a change in pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-07 Thread GitBox
lw309637554 commented on a change in pull request #2765: URL: https://github.com/apache/hudi/pull/2765#discussion_r608395924 ## File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala ## @@ -49,7 +51,48 @@ object AvroConversionUtils {

[GitHub] [hudi] lw309637554 commented on a change in pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-07 Thread GitBox
lw309637554 commented on a change in pull request #2765: URL: https://github.com/apache/hudi/pull/2765#discussion_r608395652 ## File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala ## @@ -49,7 +51,48 @@ object AvroConversionUtils {

[jira] [Updated] (HUDI-1551) Support Partition with BigDecimal/Integer field

2021-04-07 Thread Chanh Le (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chanh Le updated HUDI-1551: --- Description: In my data the time indicator field is in BigDecimal/Integer -> due to trading data related so

[jira] [Updated] (HUDI-1551) Support Partition with BigDecimal/Integer field

2021-04-07 Thread Chanh Le (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chanh Le updated HUDI-1551: --- Fix Version/s: (was: 0.7.0) > Support Partition with BigDecimal/Integer field >

[jira] [Closed] (HUDI-1767) Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink

2021-04-07 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1767. -- Resolution: Fixed d035fcbb3cf7aa33bafdd902a5e6a76f549e7ee1 > Add setter to HoodieKey and HoodieRecordLocation

[jira] [Assigned] (HUDI-1767) Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink

2021-04-07 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-1767: -- Assignee: 谢波 > Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE > performance

[jira] [Commented] (HUDI-1767) Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink

2021-04-07 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17315994#comment-17315994 ] vinoyang commented on HUDI-1767: [~hiscat] I have given you Jira contributor permission. > Add setter to

[jira] [Assigned] (HUDI-1772) HoodieFileGroupId compareTo logical error(fileId self compare)

2021-04-07 Thread Jira
[ https://issues.apache.org/jira/browse/HUDI-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 谢波 reassigned HUDI-1772: Assignee: 谢波 > HoodieFileGroupId compareTo logical error(fileId self compare) >

[jira] [Created] (HUDI-1772) HoodieFileGroupId compareTo logical error(fileId self compare)

2021-04-07 Thread Jira
谢波 created HUDI-1772: Summary: HoodieFileGroupId compareTo logical error(fileId self compare) Key: HUDI-1772 URL: https://issues.apache.org/jira/browse/HUDI-1772 Project: Apache Hudi Issue Type: Bug

[GitHub] [hudi] MyLanPangzi opened a new pull request #2780: bug fix. HoodieFileGroupId self-compare

2021-04-07 Thread GitBox
MyLanPangzi opened a new pull request #2780: URL: https://github.com/apache/hudi/pull/2780 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of

[GitHub] [hudi] ztcheck removed a comment on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
ztcheck removed a comment on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-814771640 > @ztcheck Are you using the `hudi-hive-sync-bundle` to execute the script `run_sync_tool .sh` ? > > Can you please provide what `HUDI_HIVE_UBER_JAR` is in your env ?

[GitHub] [hudi] ztcheck commented on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
ztcheck commented on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-814771640 > @ztcheck Are you using the `hudi-hive-sync-bundle` to execute the script `run_sync_tool .sh` ? > > Can you please provide what `HUDI_HIVE_UBER_JAR` is in your env ?

[GitHub] [hudi] ztcheck edited a comment on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
ztcheck edited a comment on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-814772442 @n3nash, Yes,`hudi-hive-sync-bundle` already in the script `run_sync_tool .sh` . I use the default value `HUDI_HIVE_UBER_JAR` in the script.Just like '

[GitHub] [hudi] ztcheck edited a comment on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
ztcheck edited a comment on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-800020976 My environment is k8s. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hudi] ztcheck edited a comment on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
ztcheck edited a comment on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-814772442 @n3nash, Yes,`hudi-hive-sync-bundle` already in the script `run_sync_tool .sh` . I use the default value `HUDI_HIVE_UBER_JAR` in the script.Such like '

[GitHub] [hudi] ztcheck edited a comment on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
ztcheck edited a comment on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-814772442 @n3nash, Yes,`hudi-hive-sync-bundle` already in the script `run_sync_tool .sh` . I use the default value `HUDI_HIVE_UBER_JAR` in the script, such like '

[hudi] branch master updated (dadd081 -> f4f9dd9)

2021-04-07 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from dadd081 [HUDI-1751] DeltaStreamer print many unnecessary warn log (#2754) add f4f9dd9 [HUDI-1772]

[GitHub] [hudi] yanghua merged pull request #2780: [HUDI-1772] HoodieFileGroupId compareTo logical error(fileId self compare)

2021-04-07 Thread GitBox
yanghua merged pull request #2780: URL: https://github.com/apache/hudi/pull/2780 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[jira] [Closed] (HUDI-1772) HoodieFileGroupId compareTo logical error(fileId self compare)

2021-04-07 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1772. -- Resolution: Fixed f4f9dd9d83a6a852c0e733802c6c49747cde5531 > HoodieFileGroupId compareTo logical error(fileId

[hudi] branch master updated: [HUDI-1773] HoodieFileGroup code optimize (#2781)

2021-04-07 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 3a926aa [HUDI-1773] HoodieFileGroup code

[GitHub] [hudi] yanghua merged pull request #2781: [HUDI-1773] HoodieFileGroup code optimize

2021-04-07 Thread GitBox
yanghua merged pull request #2781: URL: https://github.com/apache/hudi/pull/2781 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[jira] [Updated] (HUDI-1773) HoodieFileGroup code optimize

2021-04-07 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-1773: --- Fix Version/s: 0.9.0 > HoodieFileGroup code optimize > - > > Key:

[jira] [Closed] (HUDI-1773) HoodieFileGroup code optimize

2021-04-07 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1773. -- Resolution: Done 3a926aacf6552fc06005db4a7880a233db904330 > HoodieFileGroup code optimize >

[GitHub] [hudi] ztcheck commented on issue #2680: [SUPPORT]Hive sync error by using run_sync_tool.sh

2021-04-07 Thread GitBox
ztcheck commented on issue #2680: URL: https://github.com/apache/hudi/issues/2680#issuecomment-814772442 @n3nash, Yes,`hudi-hive-sync-bundle` already in the script `run_sync_tool .sh` . I use the default value `HUDI_HIVE_UBER_JAR` in the script.Just like `HUDI_HIVE_UBER_JAR=`ls -c

[GitHub] [hudi] lw309637554 commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
lw309637554 commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r608501941 ## File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java ## @@ -1013,26 +1014,22 @@ public void

[GitHub] [hudi] li36909 commented on pull request #2754: [HUDI-1751] Remove irrelevant properties from passing to kafkaConsumer which in turn prints lot of warn logs

2021-04-07 Thread GitBox
li36909 commented on pull request #2754: URL: https://github.com/apache/hudi/pull/2754#issuecomment-814932095 @n3nash @pratyakshsharma thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[jira] [Commented] (HUDI-1674) add partition level delete DOC or example

2021-04-07 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316432#comment-17316432 ] sivabalan narayanan commented on HUDI-1674: --- [~309637554]: we are yet to add this operation to

[GitHub] [hudi] li36909 commented on pull request #2753: [HUDI-1750] Fail to load user's class if user move hudi-spark-bundle jar into spark classpath

2021-04-07 Thread GitBox
li36909 commented on pull request #2753: URL: https://github.com/apache/hudi/pull/2753#issuecomment-814930600 @nsivabalan thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] BenjMaq commented on issue #2399: [SUPPORT] Hudi deletes not being properly commited

2021-04-07 Thread GitBox
BenjMaq commented on issue #2399: URL: https://github.com/apache/hudi/issues/2399#issuecomment-814990246 Just want to add that I faced the same issue. For me, the problem was related to the option `.option(DataSourceWriteOptions.KEYGENERATOR_CLASS_OPT_KEY,

[GitHub] [hudi] codecov-io edited a comment on pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-07 Thread GitBox
codecov-io edited a comment on pull request #2765: URL: https://github.com/apache/hudi/pull/2765#issuecomment-813008111 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [hudi] li36909 commented on pull request #2752: [HUDI-1749] Clean/Compaction/Rollback command maybe never exit when operation fail

2021-04-07 Thread GitBox
li36909 commented on pull request #2752: URL: https://github.com/apache/hudi/pull/2752#issuecomment-814931307 @n3nash thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[jira] [Created] (HUDI-1774) Add support or delete_partition with spark ds

2021-04-07 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-1774: - Summary: Add support or delete_partition with spark ds Key: HUDI-1774 URL: https://issues.apache.org/jira/browse/HUDI-1774 Project: Apache Hudi

[GitHub] [hudi] garyli1019 opened a new pull request #2783: [DOCS]Add docs for 0.8.0 release

2021-04-07 Thread GitBox
garyli1019 opened a new pull request #2783: URL: https://github.com/apache/hudi/pull/2783 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the

[GitHub] [hudi] nsivabalan commented on issue #2399: [SUPPORT] Hudi deletes not being properly commited

2021-04-07 Thread GitBox
nsivabalan commented on issue #2399: URL: https://github.com/apache/hudi/issues/2399#issuecomment-815011207 btw, we have filed a feature request to support reusing existing hudi configs https://issues.apache.org/jira/browse/HUDI-1640 -- This is an automated message from the Apache Git

[jira] [Updated] (HUDI-73) Support vanilla Avro Kafka Source in HoodieDeltaStreamer

2021-04-07 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-73?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-73: Fix Version/s: (was: 0.8.0) 0.9.0 > Support vanilla Avro Kafka Source in HoodieDeltaStreamer >

[GitHub] [hudi] codecov-io edited a comment on pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support

2021-04-07 Thread GitBox
codecov-io edited a comment on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-792430670 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=h1) Report > Merging [#2645](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=desc) (647e322) into

[jira] [Assigned] (HUDI-1760) Incorrect Documentation for HoodieWriteConfigs

2021-04-07 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li reassigned HUDI-1760: - Assignee: Gary Li > Incorrect Documentation for HoodieWriteConfigs >

[jira] [Assigned] (HUDI-1774) Add support or delete_partition with spark ds

2021-04-07 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-1774: - Assignee: liwei > Add support or delete_partition with spark ds >

[GitHub] [hudi] nsivabalan commented on issue #2743: Do we have any TTL mechanism in Hudi?

2021-04-07 Thread GitBox
nsivabalan commented on issue #2743: URL: https://github.com/apache/hudi/issues/2743#issuecomment-815015923 @lw309637554 @satishkotha : fyi we are yet to add spark ds support for this "delete_partition" operation. -- This is an automated message from the Apache Git Service. To respond

[jira] [Updated] (HUDI-1740) insert_overwrite_table and insert_overwrite first replacecommit has empty partitionToReplaceFileIds

2021-04-07 Thread Susu Dong (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Susu Dong updated HUDI-1740: Description: insert_overwrite_table and insert_overwrite first replacecommit has empty

[GitHub] [hudi] ssdong opened a new pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
ssdong opened a new pull request #2784: URL: https://github.com/apache/hudi/pull/2784 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the

[GitHub] [hudi] kvallala commented on issue #2528: [SUPPORT] Spark read hudi data from hive (metastore)

2021-04-07 Thread GitBox
kvallala commented on issue #2528: URL: https://github.com/apache/hudi/issues/2528#issuecomment-815182803 We are having the same issue. It works with `spark.sql.hive.convertMetastoreParquet=false` when querying Hudi table from spark session, but see duplicates when querying through

[GitHub] [hudi] nsivabalan commented on a change in pull request #2783: [DOCS]Add docs for 0.8.0 release

2021-04-07 Thread GitBox
nsivabalan commented on a change in pull request #2783: URL: https://github.com/apache/hudi/pull/2783#discussion_r608886318 ## File path: docs/_docs/0.8.0/1_1_spark_quick_start_guide.md ## @@ -0,0 +1,530 @@ +--- +version: 0.8.0 +title: "Quick-Start Guide" +permalink:

[GitHub] [hudi] codecov-io commented on pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
codecov-io commented on pull request #2784: URL: https://github.com/apache/hudi/pull/2784#issuecomment-815166346 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2784?src=pr=h1) Report > Merging [#2784](https://codecov.io/gh/apache/hudi/pull/2784?src=pr=desc) (5572b9f) into

[GitHub] [hudi] ze-engineering-code-challenge commented on pull request #2665: [HUDI-1160] Support update partial fields for CoW table

2021-04-07 Thread GitBox
ze-engineering-code-challenge commented on pull request #2665: URL: https://github.com/apache/hudi/pull/2665#issuecomment-815168500 Hello @liujinhui1994 Should I enable any option to work? Im trying to do an upsert in a Hudi table with 0.8.0 version and didn't work :(

[GitHub] [hudi] vingov commented on pull request #2747: [HUDI-1743] Added support for SqlFileBasedTransformer

2021-04-07 Thread GitBox
vingov commented on pull request #2747: URL: https://github.com/apache/hudi/pull/2747#issuecomment-815167427 @yanghua - I don't see the unit tests for the existing transformers except for two functions, I don't have time now to write unit tests, can I handle it in a separate pull request

[jira] [Assigned] (HUDI-1739) insert_overwrite_table and insert_overwrite create empty replacecommit.requested file which breaks archival

2021-04-07 Thread Susu Dong (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Susu Dong reassigned HUDI-1739: --- Assignee: Susu Dong > insert_overwrite_table and insert_overwrite create empty >

[GitHub] [hudi] ssdong commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-04-07 Thread GitBox
ssdong commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-815156685 Hi @satishkotha @jsbali ! I've created the pull request for this issue. Had observed more when going down the road and I've tried my best to clarify them and hopefully had written a

[GitHub] [hudi] stackfun commented on issue #2771: [SUPPORT] Log files are not compacted

2021-04-07 Thread GitBox
stackfun commented on issue #2771: URL: https://github.com/apache/hudi/issues/2771#issuecomment-815292886 Setting the "hoodie.compaction.target.io" config worked like a charm. Thanks a lot! -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hudi] stackfun closed issue #2771: [SUPPORT] Log files are not compacted

2021-04-07 Thread GitBox
stackfun closed issue #2771: URL: https://github.com/apache/hudi/issues/2771 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] jintaoguan commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
jintaoguan commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r609161570 ## File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java ## @@ -1013,26 +1014,22 @@ public void

[GitHub] [hudi] satishkotha commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
satishkotha commented on a change in pull request #2784: URL: https://github.com/apache/hudi/pull/2784#discussion_r609094597 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/MetadataConversionUtils.java ## @@ -105,14 +114,15 @@ public

[GitHub] [hudi] rubenssoto commented on issue #2770: [SUPPORT] How column _hoodie_is_deleted works?

2021-04-07 Thread GitBox
rubenssoto commented on issue #2770: URL: https://github.com/apache/hudi/issues/2770#issuecomment-815296739 @nsivabalan I think the error in on my side. I didn't filter the deleted records on the first batch, it could be a great feature to Hudi in the future. -- This is an

[GitHub] [hudi] nsivabalan commented on issue #2770: [SUPPORT] How column _hoodie_is_deleted works?

2021-04-07 Thread GitBox
nsivabalan commented on issue #2770: URL: https://github.com/apache/hudi/issues/2770#issuecomment-815315164 Sorry, what feature you are looking for. can you please clarify. hudi automatically deletes those records which has "_hoodie_is_deleted" set to true. in other words, if you have a

[GitHub] [hudi] satishkotha commented on pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
satishkotha commented on pull request #2784: URL: https://github.com/apache/hudi/pull/2784#issuecomment-815291682 @ssdong thanks for bringing this up and contributing. I added some comments, please take a look. Also, looks like there are some CI failures. Please fix those as well. --

[GitHub] [hudi] zherenyu831 commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
zherenyu831 commented on a change in pull request #2784: URL: https://github.com/apache/hudi/pull/2784#discussion_r609200500 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java ## @@ -245,7 +245,7 @@ public final void

[GitHub] [hudi] lw309637554 commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
lw309637554 commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r609215373 ## File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java ## @@ -1013,26 +1014,22 @@ public void

[GitHub] [hudi] susudong commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
susudong commented on a change in pull request #2784: URL: https://github.com/apache/hudi/pull/2784#discussion_r609229811 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/MetadataConversionUtils.java ## @@ -72,9 +76,14 @@ public static

[GitHub] [hudi] lw309637554 commented on pull request #2765: [HUDI-1716]: Resolving default values for schema from dataframe

2021-04-07 Thread GitBox
lw309637554 commented on pull request #2765: URL: https://github.com/apache/hudi/pull/2765#issuecomment-815405833 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [hudi] ssdong commented on pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-07 Thread GitBox
ssdong commented on pull request #2784: URL: https://github.com/apache/hudi/pull/2784#issuecomment-815413234 @satishkotha Thank you for your review! I’ll take a look when I get back. Currently on a day trip.  Basically, I wanna stop the abuse of `REQUESTED` here, at least for the

[GitHub] [hudi] codecov-io edited a comment on pull request #2645: [HUDI-1659] Basic Implementation Of Spark Sql Support

2021-04-07 Thread GitBox
codecov-io edited a comment on pull request #2645: URL: https://github.com/apache/hudi/pull/2645#issuecomment-792430670 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=h1) Report > Merging [#2645](https://codecov.io/gh/apache/hudi/pull/2645?src=pr=desc) (46516da) into

[GitHub] [hudi] yanghua commented on pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-04-07 Thread GitBox
yanghua commented on pull request #2325: URL: https://github.com/apache/hudi/pull/2325#issuecomment-815419885 > @wangxianghu: It's OK now. Thanks for your patience, I will do a final check soon. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] hddong commented on pull request #1946: [HUDI-1176]Upgrade tp log4j2

2021-04-07 Thread GitBox
hddong commented on pull request #1946: URL: https://github.com/apache/hudi/pull/1946#issuecomment-815376490 @wangxianghu: had upgrade to `2.13.3` and fix the warning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[jira] [Resolved] (HUDI-1750) Fail to load user's class if user move hudi-spark-bundle_2.11-0.7.0.jar into spark classpath

2021-04-07 Thread lrz (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lrz resolved HUDI-1750. --- Resolution: Fixed > Fail to load user's class if user move hudi-spark-bundle_2.11-0.7.0.jar into > spark classpath >

[GitHub] [hudi] lw309637554 commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
lw309637554 commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r609224532 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/ClusteringCommand.java ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] lw309637554 commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
lw309637554 commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r609227046 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/ClusteringCommand.java ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] lw309637554 commented on a change in pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-07 Thread GitBox
lw309637554 commented on a change in pull request #2773: URL: https://github.com/apache/hudi/pull/2773#discussion_r609227257 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/ClusteringCommand.java ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] n3nash commented on issue #2743: Do we have any TTL mechanism in Hudi?

2021-04-07 Thread GitBox
n3nash commented on issue #2743: URL: https://github.com/apache/hudi/issues/2743#issuecomment-815456351 @aditiwari01 I think you mentioned 2 issues here 1. Record level TTL -> We don't have such a feature in Hudi. Like others have pointed out, using the

  1   2   >