[GitHub] [hudi] sathyaprakashg commented on pull request #2012: [HUDI-1129] Deltastreamer Add support for schema evolution

2020-09-17 Thread GitBox
sathyaprakashg commented on pull request #2012: URL: https://github.com/apache/hudi/pull/2012#issuecomment-694081153 @n3nash Changes are done. Please review it when you get time This is an automated message from the Apache

[GitHub] [hudi] vinothchandar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

2020-09-17 Thread GitBox
vinothchandar commented on a change in pull request #2064: URL: https://github.com/apache/hudi/pull/2064#discussion_r488962216 ## File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java ## @@ -194,6 +196,35 @@ public boolean

[GitHub] [hudi] xushiyan opened a new pull request #2094: [HUDI-995] Migrate HoodieTestUtils APIs to HoodieTestTable

2020-09-17 Thread GitBox
xushiyan opened a new pull request #2094: URL: https://github.com/apache/hudi/pull/2094 ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc

[GitHub] [hudi] WTa-hash commented on issue #2057: [SUPPORT] AWSDmsAvroPayload not processing Deletes correctly + IOException when reading log file

2020-09-17 Thread GitBox
WTa-hash commented on issue #2057: URL: https://github.com/apache/hudi/issues/2057#issuecomment-694437332 > @umehrot2 Could the IOException be due to #2089 ? I'm not entirely sure if it's related to this issue as the steps to reproduce is different, but the thing I see in common is

[GitHub] [hudi] n3nash commented on issue #2089: Reading MOR Tables - Not Working

2020-09-17 Thread GitBox
n3nash commented on issue #2089: URL: https://github.com/apache/hudi/issues/2089#issuecomment-694451216 Hmm, seems like some misconfiguration. Did you try `s3://`, from the logs it looks like it's defaulting to /mnt ? This

[jira] [Commented] (HUDI-995) Organize test utils methods and classes

2020-09-17 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197937#comment-17197937 ] Raymond Xu commented on HUDI-995: - Working on the 3rd point in the description. After that, we may close

[GitHub] [hudi] xushiyan commented on a change in pull request #2094: [HUDI-995] Migrate HoodieTestUtils APIs to HoodieTestTable

2020-09-17 Thread GitBox
xushiyan commented on a change in pull request #2094: URL: https://github.com/apache/hudi/pull/2094#discussion_r490540439 ## File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java ## @@ -268,16 +292,16 @@ public boolean

[GitHub] [hudi] satishkotha commented on a change in pull request #2048: [HUDI-1072][WIP] Introduce REPLACE top level action

2020-09-17 Thread GitBox
satishkotha commented on a change in pull request #2048: URL: https://github.com/apache/hudi/pull/2048#discussion_r490461331 ## File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java ## @@ -301,6 +304,61 @@ private void

[GitHub] [hudi] satishkotha commented on a change in pull request #2048: [HUDI-1072][WIP] Introduce REPLACE top level action

2020-09-17 Thread GitBox
satishkotha commented on a change in pull request #2048: URL: https://github.com/apache/hudi/pull/2048#discussion_r490461178 ## File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java ## @@ -301,6 +304,61 @@ private void

[GitHub] [hudi] satishkotha commented on a change in pull request #2048: [HUDI-1072][WIP] Introduce REPLACE top level action

2020-09-17 Thread GitBox
satishkotha commented on a change in pull request #2048: URL: https://github.com/apache/hudi/pull/2048#discussion_r490476688 ## File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java ## @@ -301,6 +304,61 @@ private void

[GitHub] [hudi] satishkotha commented on a change in pull request #2048: [HUDI-1072][WIP] Introduce REPLACE top level action

2020-09-17 Thread GitBox
satishkotha commented on a change in pull request #2048: URL: https://github.com/apache/hudi/pull/2048#discussion_r490491745 ## File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java ## @@ -301,6 +304,61 @@ private void

[GitHub] [hudi] satishkotha commented on a change in pull request #2048: [HUDI-1072][WIP] Introduce REPLACE top level action

2020-09-17 Thread GitBox
satishkotha commented on a change in pull request #2048: URL: https://github.com/apache/hudi/pull/2048#discussion_r490491745 ## File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java ## @@ -301,6 +304,61 @@ private void

[jira] [Updated] (HUDI-995) Organize test utils methods and classes

2020-09-17 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-995: Description: * Move test utils classes to hudi-common where appropriate, e.g. TestRawTripPayload,

[GitHub] [hudi] bvaradar commented on pull request #1650: [HUDI-541]: replaced dataFile/df with baseFile/bf throughout code base

2020-09-17 Thread GitBox
bvaradar commented on pull request #1650: URL: https://github.com/apache/hudi/pull/1650#issuecomment-694493239 @pratyakshsharma : As there are already many large refactorings/features going on, Shall we close this PR for now and may be take a relook for next release. Let me know.

[GitHub] [hudi] vinothchandar commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

2020-09-17 Thread GitBox
vinothchandar commented on pull request #2064: URL: https://github.com/apache/hudi/pull/2064#issuecomment-694538029 Also see a lot of this polluting the test logs. ``` test-trip-table_metadata.timer.deltacommit count = 2 mean rate = 0.00 calls/second

[GitHub] [hudi] satishkotha commented on a change in pull request #2048: [HUDI-1072][WIP] Introduce REPLACE top level action

2020-09-17 Thread GitBox
satishkotha commented on a change in pull request #2048: URL: https://github.com/apache/hudi/pull/2048#discussion_r490621340 ## File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java ## @@ -301,6 +304,61 @@ private void

[GitHub] [hudi] bvaradar commented on pull request #2069: [WIP][HUDI-945] Cleanup Spillable map files eagerly for DiskBasedMap

2020-09-17 Thread GitBox
bvaradar commented on pull request #2069: URL: https://github.com/apache/hudi/pull/2069#issuecomment-694494713 @nbalajee : Since this is a critical part w.r.t merge performance and you are familiar with it, can you please take a look and review this. I will also be reviewing this.

[GitHub] [hudi] bvaradar commented on a change in pull request #2048: [HUDI-1072][WIP] Introduce REPLACE top level action

2020-09-17 Thread GitBox
bvaradar commented on a change in pull request #2048: URL: https://github.com/apache/hudi/pull/2048#discussion_r490589908 ## File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java ## @@ -301,6 +304,61 @@ private void

[jira] [Commented] (HUDI-867) Graphite metrics are throwing IllegalArgumentException on continuous mode

2020-09-17 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198009#comment-17198009 ] Raymond Xu commented on HUDI-867: - [~Pratyaksh] this can be resolved and closed right? > Graphite metrics

[GitHub] [hudi] vinothchandar commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

2020-09-17 Thread GitBox
vinothchandar commented on pull request #2064: URL: https://github.com/apache/hudi/pull/2064#issuecomment-694537286 @prashantwason the tests are exceeding limit/failing, after fixing checkstyle issues also. Can you please check?

[GitHub] [hudi] umehrot2 commented on pull request #2046: [HUDI-1230] Fix for preventing MOR datasource jobs from hanging via spark-submit

2020-09-17 Thread GitBox
umehrot2 commented on pull request #2046: URL: https://github.com/apache/hudi/pull/2046#issuecomment-694115738 Added test @bvaradar . Sorry got a bit late on this. This is an automated message from the Apache Git Service. To

[GitHub] [hudi] pratyakshsharma commented on pull request #1558: [HUDI-796]: added deduping logic for upserts case

2020-09-17 Thread GitBox
pratyakshsharma commented on pull request #1558: URL: https://github.com/apache/hudi/pull/1558#issuecomment-694142029 @yanghua Please take a pass :) This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer

2020-09-17 Thread GitBox
pratyakshsharma commented on a change in pull request #2073: URL: https://github.com/apache/hudi/pull/2073#discussion_r490165008 ## File path: docs/_posts/2020-08-22-ingest-multiple-tables-using-hudi.md ## @@ -0,0 +1,101 @@ +--- +title: "Ingest multiple tables using Hudi"

[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer

2020-09-17 Thread GitBox
pratyakshsharma commented on a change in pull request #2073: URL: https://github.com/apache/hudi/pull/2073#discussion_r490165732 ## File path: docs/_posts/2020-08-22-ingest-multiple-tables-using-hudi.md ## @@ -0,0 +1,101 @@ +--- +title: "Ingest multiple tables using Hudi"

[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer

2020-09-17 Thread GitBox
pratyakshsharma commented on a change in pull request #2073: URL: https://github.com/apache/hudi/pull/2073#discussion_r490163062 ## File path: docs/_docs/2_2_writing_data.md ## @@ -210,6 +210,8 @@ Sample config files for table wise overridden properties can be found under

[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer

2020-09-17 Thread GitBox
pratyakshsharma commented on a change in pull request #2073: URL: https://github.com/apache/hudi/pull/2073#discussion_r490163062 ## File path: docs/_docs/2_2_writing_data.md ## @@ -210,6 +210,8 @@ Sample config files for table wise overridden properties can be found under

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2046: [HUDI-1230] Fix for preventing MOR datasource jobs from hanging via spark-submit

2020-09-17 Thread GitBox
codecov-commenter edited a comment on pull request #2046: URL: https://github.com/apache/hudi/pull/2046#issuecomment-694581513 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2046?src=pr=h1) Report > Merging [#2046](https://codecov.io/gh/apache/hudi/pull/2046?src=pr=desc) into

[GitHub] [hudi] codecov-commenter commented on pull request #2046: [HUDI-1230] Fix for preventing MOR datasource jobs from hanging via spark-submit

2020-09-17 Thread GitBox
codecov-commenter commented on pull request #2046: URL: https://github.com/apache/hudi/pull/2046#issuecomment-694581513 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2046?src=pr=h1) Report > Merging [#2046](https://codecov.io/gh/apache/hudi/pull/2046?src=pr=desc) into

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2046: [HUDI-1230] Fix for preventing MOR datasource jobs from hanging via spark-submit

2020-09-17 Thread GitBox
codecov-commenter edited a comment on pull request #2046: URL: https://github.com/apache/hudi/pull/2046#issuecomment-694581513 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2046?src=pr=h1) Report > Merging [#2046](https://codecov.io/gh/apache/hudi/pull/2046?src=pr=desc) into

[jira] [Commented] (HUDI-995) Organize test utils methods and classes

2020-09-17 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198065#comment-17198065 ] vinoyang commented on HUDI-995: --- bq. Working on the 3rd point in the description. After that, we may close

[GitHub] [hudi] yanghua commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer

2020-09-17 Thread GitBox
yanghua commented on a change in pull request #2073: URL: https://github.com/apache/hudi/pull/2073#discussion_r490660995 ## File path: content/blog/ingest-multiple-tables-using-hudi/index.html ## @@ -0,0 +1,233 @@ + Review comment: Hi, It seems we do not need the

[GitHub] [hudi] nsivabalan commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer

2020-09-17 Thread GitBox
nsivabalan commented on a change in pull request #2073: URL: https://github.com/apache/hudi/pull/2073#discussion_r490670265 ## File path: docs/_posts/2020-08-22-ingest-multiple-tables-using-hudi.md ## @@ -0,0 +1,104 @@ +--- +title: "Ingest multiple tables using Hudi" +excerpt:

[hudi] branch master updated: [HUDI-1230] Fix for preventing MOR datasource jobs from hanging via spark-submit (#2046)

2020-09-17 Thread vbalaji
This is an automated email from the ASF dual-hosted git repository. vbalaji pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new bf65269 [HUDI-1230] Fix for preventing MOR

[GitHub] [hudi] bvaradar merged pull request #2046: [HUDI-1230] Fix for preventing MOR datasource jobs from hanging via spark-submit

2020-09-17 Thread GitBox
bvaradar merged pull request #2046: URL: https://github.com/apache/hudi/pull/2046 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [hudi] pratyakshsharma commented on pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer

2020-09-17 Thread GitBox
pratyakshsharma commented on pull request #2073: URL: https://github.com/apache/hudi/pull/2073#issuecomment-694335686 @yanghua @bhasudha Please take a pass. I am not sure if I added the index.html in proper form or not. Please suggest.

[GitHub] [hudi] xushiyan merged pull request #2079: [HUDI-995] Use HoodieTestTable in more classes

2020-09-17 Thread GitBox
xushiyan merged pull request #2079: URL: https://github.com/apache/hudi/pull/2079 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[hudi] branch master updated (581d540 -> 3201665)

2020-09-17 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 581d540 [HUDI-1143] Change timestamp field in HoodieTestDataGenerator from double to long add 3201665

[hudi] branch master updated (581d540 -> 3201665)

2020-09-17 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 581d540 [HUDI-1143] Change timestamp field in HoodieTestDataGenerator from double to long add 3201665

[GitHub] [hudi] pratyakshsharma commented on a change in pull request #2073: [HUDI-769] Added blog for HoodieMultiTableDeltaStreamer

2020-09-17 Thread GitBox
pratyakshsharma commented on a change in pull request #2073: URL: https://github.com/apache/hudi/pull/2073#discussion_r490166850 ## File path: docs/_posts/2020-08-22-ingest-multiple-tables-using-hudi.md ## @@ -0,0 +1,101 @@ +--- +title: "Ingest multiple tables using Hudi"

[GitHub] [hudi] harishchanderramesh commented on issue #2089: Reading MOR Tables - Not Working

2020-09-17 Thread GitBox
harishchanderramesh commented on issue #2089: URL: https://github.com/apache/hudi/issues/2089#issuecomment-694364558 Hi, Sorry for the delay in response. I Tried with `s3://` as @umehrot2 suggested and got below error. ``` Traceback (most recent call last): File