[GitHub] [hudi] vinothchandar closed pull request #2395: [DO NOT MERGE] [RFC-15] Temporary PR to aid rebasing process

2021-01-06 Thread GitBox
vinothchandar closed pull request #2395: URL: https://github.com/apache/hudi/pull/2395 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[jira] [Resolved] (HUDI-1513) Move metadata syncing to a preWrite() method, away from WriteClient constructor

2021-01-06 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar resolved HUDI-1513. -- Resolution: Fixed > Move metadata syncing to a preWrite() method, away from WriteClient >

[jira] [Resolved] (HUDI-1504) Allow log files added as a part of restore to be synced to metadata table

2021-01-06 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar resolved HUDI-1504. -- Resolution: Fixed > Allow log files added as a part of restore to be synced to metadata table >

[jira] [Updated] (HUDI-1504) Allow log files added as a part of restore to be synced to metadata table

2021-01-06 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1504: - Status: Patch Available (was: In Progress) > Allow log files added as a part of restore to be

[jira] [Updated] (HUDI-1504) Allow log files added as a part of restore to be synced to metadata table

2021-01-06 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1504: - Status: Closed (was: Patch Available) > Allow log files added as a part of restore to be synced

[jira] [Reopened] (HUDI-1504) Allow log files added as a part of restore to be synced to metadata table

2021-01-06 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reopened HUDI-1504: -- > Allow log files added as a part of restore to be synced to metadata table >

[GitHub] [hudi] vinothchandar commented on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
vinothchandar commented on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755937041 @umehrot2 should we first merge this and do the actual fixes on top as a separarte PR? if so, please feel free to land this

[GitHub] [hudi] vinothchandar commented on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
vinothchandar commented on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755936804 @yanghua I agree with you mostly. lets rethink our structure more holsitically. This is an automated

[hudi] branch master updated (b593f10 -> 5ff8e88)

2021-01-06 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from b593f10 [MINOR] Rename unit test package of hudi-spark3 from scala to java (#2411) add 5ff8e88 [HUDI-1513]

[GitHub] [hudi] vinothchandar merged pull request #2413: [HUDI-1513] Introduce WriteClient#preWrite() and relocate metadata table syncing

2021-01-06 Thread GitBox
vinothchandar merged pull request #2413: URL: https://github.com/apache/hudi/pull/2413 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [hudi] codecov-io commented on pull request #2413: [HUDI-1513] Introduce WriteClient#preWrite() and relocate metadata table syncing

2021-01-06 Thread GitBox
codecov-io commented on pull request #2413: URL: https://github.com/apache/hudi/pull/2413#issuecomment-755922887 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2413?src=pr=h1) Report > Merging [#2413](https://codecov.io/gh/apache/hudi/pull/2413?src=pr=desc) (f9992fd) into

[GitHub] [hudi] vinothchandar commented on a change in pull request #2413: [HUDI-1513] Introduce WriteClient#preWrite() and relocate metadata table syncing

2021-01-06 Thread GitBox
vinothchandar commented on a change in pull request #2413: URL: https://github.com/apache/hudi/pull/2413#discussion_r553137351 ## File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/compact/TestAsyncCompaction.java ## @@ -50,7 +51,9 @@ public

[GitHub] [hudi] vinothchandar commented on a change in pull request #2413: [HUDI-1513] Introduce WriteClient#preWrite() and relocate metadata table syncing

2021-01-06 Thread GitBox
vinothchandar commented on a change in pull request #2413: URL: https://github.com/apache/hudi/pull/2413#discussion_r553136559 ## File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/compact/TestAsyncCompaction.java ## @@ -50,7 +51,9 @@ public

[GitHub] [hudi] codecov-io edited a comment on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
codecov-io edited a comment on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755158309 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] codecov-io edited a comment on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
codecov-io edited a comment on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755158309 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2410?src=pr=h1) Report > Merging [#2410](https://codecov.io/gh/apache/hudi/pull/2410?src=pr=desc) (0e81d6e) into

[GitHub] [hudi] rmpifer commented on a change in pull request #2413: [HUDI-1513] Introduce WriteClient#preWrite() and relocate metadata table syncing

2021-01-06 Thread GitBox
rmpifer commented on a change in pull request #2413: URL: https://github.com/apache/hudi/pull/2413#discussion_r553132216 ## File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/compact/TestAsyncCompaction.java ## @@ -50,7 +51,9 @@ public class

[GitHub] [hudi] Karl-WangSK opened a new pull request #2260: [HUDI-1381] Schedule compaction based on time elapsed

2021-01-06 Thread GitBox
Karl-WangSK opened a new pull request #2260: URL: https://github.com/apache/hudi/pull/2260 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of

[GitHub] [hudi] codecov-io edited a comment on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
codecov-io edited a comment on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755158309 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2410?src=pr=h1) Report > Merging [#2410](https://codecov.io/gh/apache/hudi/pull/2410?src=pr=desc) (0e81d6e) into

[GitHub] [hudi] wosow edited a comment on issue #2409: [SUPPORT] Spark structured Streaming writes to Hudi and synchronizes Hive to create only read-optimized tables without creating real-time tables

2021-01-06 Thread GitBox
wosow edited a comment on issue #2409: URL: https://github.com/apache/hudi/issues/2409#issuecomment-755894869 > It is indeed a MOR table.Can you check your driver logs. You might find some exceptions around registering _rt table. You can look for logs around the log message > >

[GitHub] [hudi] wosow edited a comment on issue #2409: [SUPPORT] Spark structured Streaming writes to Hudi and synchronizes Hive to create only read-optimized tables without creating real-time tables

2021-01-06 Thread GitBox
wosow edited a comment on issue #2409: URL: https://github.com/apache/hudi/issues/2409#issuecomment-755894869 > It is indeed a MOR table.Can you check your driver logs. You might find some exceptions around registering _rt table. You can look for logs around the log message > >

[GitHub] [hudi] wosow commented on issue #2409: [SUPPORT] Spark structured Streaming writes to Hudi and synchronizes Hive to create only read-optimized tables without creating real-time tables

2021-01-06 Thread GitBox
wosow commented on issue #2409: URL: https://github.com/apache/hudi/issues/2409#issuecomment-755894869 > It is indeed a MOR table.Can you check your driver logs. You might find some exceptions around registering _rt table. You can look for logs around the log message > > "Trying to

[GitHub] [hudi] nsivabalan commented on a change in pull request #2111: [HUDI-1234] Insert new records regardless of small file when using insert operation

2021-01-06 Thread GitBox
nsivabalan commented on a change in pull request #2111: URL: https://github.com/apache/hudi/pull/2111#discussion_r553104712 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/UpsertPartitioner.java ## @@ -191,14 +201,29 @@ private

[GitHub] [hudi] nsivabalan commented on pull request #2111: [HUDI-1234] Insert new records regardless of small file when using insert operation

2021-01-06 Thread GitBox
nsivabalan commented on pull request #2111: URL: https://github.com/apache/hudi/pull/2111#issuecomment-755879362 @vinothchandar : I have updated the patch as requested. Mid way in fixing tests(fixed some and ensured that both old and new code path works as expected though). But in the

[GitHub] [hudi] nsivabalan commented on a change in pull request #2111: [HUDI-1234] Insert new records regardless of small file when using insert operation

2021-01-06 Thread GitBox
nsivabalan commented on a change in pull request #2111: URL: https://github.com/apache/hudi/pull/2111#discussion_r553104712 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/UpsertPartitioner.java ## @@ -191,14 +201,29 @@ private

[GitHub] [hudi] yanghua commented on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
yanghua commented on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755878176 I agree with this operation so as not to block work progress. However, I always feel that in terms of project layout, if we treat writing or reading (or call it query) equally, it

[jira] [Comment Edited] (HUDI-1462) The rt view query returns a wrong result with predicate push down

2021-01-06 Thread qian heng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260218#comment-17260218 ] qian heng edited comment on HUDI-1462 at 1/7/21, 4:21 AM: -- [~danny0405] It means

[jira] [Updated] (HUDI-1462) The rt view query returns a wrong result with predicate push down

2021-01-06 Thread qian heng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qian heng updated HUDI-1462: Description: The rt view query returns a wrong result with predicate push down. This is my query on a

[jira] [Commented] (HUDI-1462) The rt view query returns a wrong result with predicate push down

2021-01-06 Thread qian heng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260218#comment-17260218 ] qian heng commented on HUDI-1462: - [~danny0405] it means Near-Realtime (RT) table. > The rt view query

[jira] [Comment Edited] (HUDI-1462) The rt view query returns a wrong result with predicate push down

2021-01-06 Thread qian heng (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260218#comment-17260218 ] qian heng edited comment on HUDI-1462 at 1/7/21, 4:19 AM: -- [~danny0405] I means

[GitHub] [hudi] bvaradar commented on issue #2406: [SUPPORT] Deltastreamer - Property hoodie.datasource.write.partitionpath.field not found

2021-01-06 Thread GitBox
bvaradar commented on issue #2406: URL: https://github.com/apache/hudi/issues/2406#issuecomment-755871276 No worries. Should be fine to keep this open This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] bvaradar commented on issue #2409: [SUPPORT] Spark structured Streaming writes to Hudi and synchronizes Hive to create only read-optimized tables without creating real-time tables

2021-01-06 Thread GitBox
bvaradar commented on issue #2409: URL: https://github.com/apache/hudi/issues/2409#issuecomment-755870995 It is indeed a MOR table.Can you check your driver logs. You might find some exceptions around registering _rt table. You can look for logs around the log message "Trying to

[GitHub] [hudi] vinothchandar commented on pull request #2413: [HUDI-1513] Introduce WriteClient#preWrite() and relocate metadata table syncing

2021-01-06 Thread GitBox
vinothchandar commented on pull request #2413: URL: https://github.com/apache/hudi/pull/2413#issuecomment-755857098 Yes. Correct. This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hudi] pengzhiwei2018 edited a comment on pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…

2021-01-06 Thread GitBox
pengzhiwei2018 edited a comment on pull request #2334: URL: https://github.com/apache/hudi/pull/2334#issuecomment-754400800 Hi @n3nash ,Thanks for your suggestion. This PR only covers the part where HUDi writes the data, not the part where hudi reads the data. The introduced

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2350: [HUDI-1493] Fixed schema compatibility check for fields.

2021-01-06 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2350: URL: https://github.com/apache/hudi/pull/2350#discussion_r553077554 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java ## @@ -296,7 +296,7 @@ public MessageType

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2350: [HUDI-1493] Fixed schema compatibility check for fields.

2021-01-06 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2350: URL: https://github.com/apache/hudi/pull/2350#discussion_r553077554 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java ## @@ -296,7 +296,7 @@ public MessageType

[GitHub] [hudi] cdmikechen commented on issue #2100: [SUPPORT] 0.6.0 - using keytab authentication gives issues

2021-01-06 Thread GitBox
cdmikechen commented on issue #2100: URL: https://github.com/apache/hudi/issues/2100#issuecomment-755846060 @vinothchandar Yes! I deployed the timeline server separately on its own and let multiple hudi services use this timeline server. I used an unified timeline server to obtain

[GitHub] [hudi] Karl-WangSK closed pull request #2260: [HUDI-1381] Schedule compaction based on time elapsed

2021-01-06 Thread GitBox
Karl-WangSK closed pull request #2260: URL: https://github.com/apache/hudi/pull/2260 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [hudi] Karl-WangSK commented on a change in pull request #2260: [HUDI-1381] Schedule compaction based on time elapsed

2021-01-06 Thread GitBox
Karl-WangSK commented on a change in pull request #2260: URL: https://github.com/apache/hudi/pull/2260#discussion_r553075502 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java ## @@ -60,17

[jira] [Assigned] (HUDI-1332) Introduce FlinkHoodieBloomIndex to hudi-flink-client

2021-01-06 Thread Xiang Yang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Yang reassigned HUDI-1332: Assignee: Xiang Yang (was: Gary Li) > Introduce FlinkHoodieBloomIndex to hudi-flink-client >

[GitHub] [hudi] wosow commented on issue #2409: [SUPPORT] Spark structured Streaming writes to Hudi and synchronizes Hive to create only read-optimized tables without creating real-time tables

2021-01-06 Thread GitBox
wosow commented on issue #2409: URL: https://github.com/apache/hudi/issues/2409#issuecomment-755838339 > can you copy the contents of hoodie.properties of the dataset here ? hoodie.properties as follows: [hoodie.zip](https://github.com/apache/hudi/files/5779109/hoodie.zip)

[GitHub] [hudi] prashantwason commented on pull request #2413: [HUDI-1513] Introduce WriteClient#preWrite() and relocate metadata table syncing

2021-01-06 Thread GitBox
prashantwason commented on pull request #2413: URL: https://github.com/apache/hudi/pull/2413#issuecomment-755833153 Looks fine. I checked and you can verify too, there is no API in HoodieWriteClient which can be called before preWrite() and end up returning a stale version of

[jira] [Commented] (HUDI-1459) Support for handling of REPLACE instants

2021-01-06 Thread Prashant Wason (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260150#comment-17260150 ] Prashant Wason commented on HUDI-1459: -- Oh great. Yes, HUDI-1276 would be a very elegant way to fix

[jira] [Commented] (HUDI-1459) Support for handling of REPLACE instants

2021-01-06 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260145#comment-17260145 ] Vinoth Chandar commented on HUDI-1459: -- It seems like if we did [HUDI-1276], and option 2 would very

[jira] [Commented] (HUDI-1459) Support for handling of REPLACE instants

2021-01-06 Thread Prashant Wason (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260143#comment-17260143 ] Prashant Wason commented on HUDI-1459: -- [~vinoth] [~nishith29] [~satish] What are your thoughts on

[jira] [Commented] (HUDI-1459) Support for handling of REPLACE instants

2021-01-06 Thread Prashant Wason (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260142#comment-17260142 ] Prashant Wason commented on HUDI-1459: -- The replace functionality is as follows: # When

[GitHub] [hudi] vinothchandar commented on pull request #2413: [HUDI-1513] Introduce WriteClient#preWrite() and relocate metadata table syncing

2021-01-06 Thread GitBox
vinothchandar commented on pull request #2413: URL: https://github.com/apache/hudi/pull/2413#issuecomment-755817612 cc @prashantwason @rmpifer FYI This is an automated message from the Apache Git Service. To respond to the

[jira] [Updated] (HUDI-1513) Move metadata syncing to a preWrite() method, away from WriteClient constructor

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1513: - Labels: pull-request-available (was: ) > Move metadata syncing to a preWrite() method, away from

[GitHub] [hudi] vinothchandar opened a new pull request #2413: [HUDI-1513] Introduce WriteClient#preWrite() and relocate metadata table syncing

2021-01-06 Thread GitBox
vinothchandar opened a new pull request #2413: URL: https://github.com/apache/hudi/pull/2413 - Syncing to metadata table, setting operation type, starting async cleaner done in preWrite() - Fixes an issues where delete() was not starting async cleaner correctly - Fixed tests and

[GitHub] [hudi] vinothchandar commented on issue #2100: [SUPPORT] 0.6.0 - using keytab authentication gives issues

2021-01-06 Thread GitBox
vinothchandar commented on issue #2100: URL: https://github.com/apache/hudi/issues/2100#issuecomment-755814009 @cdmikechen are you deploying the timeline server separately on its own? That's interesting. Don't think anyone has tested it this way. cc @leesf as well, in case he has done

[jira] [Created] (HUDI-1513) Move metadata syncing to a preWrite() method, away from WriteClient constructor

2021-01-06 Thread Vinoth Chandar (Jira)
Vinoth Chandar created HUDI-1513: Summary: Move metadata syncing to a preWrite() method, away from WriteClient constructor Key: HUDI-1513 URL: https://issues.apache.org/jira/browse/HUDI-1513 Project:

[jira] [Updated] (HUDI-1513) Move metadata syncing to a preWrite() method, away from WriteClient constructor

2021-01-06 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1513: - Status: In Progress (was: Open) > Move metadata syncing to a preWrite() method, away from

[GitHub] [hudi] cdmikechen commented on issue #2100: [SUPPORT] 0.6.0 - using keytab authentication gives issues

2021-01-06 Thread GitBox
cdmikechen commented on issue #2100: URL: https://github.com/apache/hudi/issues/2100#issuecomment-755797895 @bhasudha I have a new problem. If the timeline server is deployed separately in the kerberized hdfs environment, and the service is called after the token refresh time of

[GitHub] [hudi] satishkotha edited a comment on issue #2346: [SUPPORT]The rt view query returns a wrong result with predicate push down.

2021-01-06 Thread GitBox
satishkotha edited a comment on issue #2346: URL: https://github.com/apache/hudi/issues/2346#issuecomment-755791465 @sumihehe I'm not able to reproduce this issue using dataset in [docker demo](https://hudi.apache.org/docs/docker_demo.html) 0: jdbc:hive2://hiveserver:1> select

[GitHub] [hudi] satishkotha commented on issue #2346: [SUPPORT]The rt view query returns a wrong result with predicate push down.

2021-01-06 Thread GitBox
satishkotha commented on issue #2346: URL: https://github.com/apache/hudi/issues/2346#issuecomment-755791465 @sumihehe I'm not able to reproduce this issue using dataset in [docker demo](https://hudi.apache.org/docs/docker_demo.html) `0: jdbc:hive2://hiveserver:1> select

[GitHub] [hudi] umehrot2 edited a comment on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
umehrot2 edited a comment on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755777542 @yanghua Thank you for the feedback and bringing up a good discussion. Me and Vinoth had a quick call on this, and we still feel that option A of moving

[GitHub] [hudi] umehrot2 commented on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
umehrot2 commented on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755777542 @yanghua Thank you for the feedback and bringing up a good discussion. Me and Vinoth had a quick call on this, and we still feel that option A of moving `HoodieEngineContext`

[GitHub] [hudi] codecov-io commented on pull request #2412: [HUDI-1512] Fix spark 2 unit tests failure with Spark 3

2021-01-06 Thread GitBox
codecov-io commented on pull request #2412: URL: https://github.com/apache/hudi/pull/2412#issuecomment-755726635 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2412?src=pr=h1) Report > Merging [#2412](https://codecov.io/gh/apache/hudi/pull/2412?src=pr=desc) (aff4d7e) into

[GitHub] [hudi] vinothchandar edited a comment on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
vinothchandar edited a comment on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755529282 This points to a deeper discussion. Right now, hudi-common is depended upon by `hudi-hadoop-mr` which has all the Hive/InputFormat code. Should that also be redone as

[GitHub] [hudi] vinothchandar edited a comment on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
vinothchandar edited a comment on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755529282 This points to a deeper discussion. Right now, hudi-common is depended upon by `hudi-hadoop-mr` which has all the Hive/InputFormat code. Should that also be redone as

[jira] [Closed] (HUDI-1507) Hive sync having issues w/ Clustering

2021-01-06 Thread satish (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] satish closed HUDI-1507. > Hive sync having issues w/ Clustering > - > > Key: HUDI-1507 >

[jira] [Resolved] (HUDI-1507) Hive sync having issues w/ Clustering

2021-01-06 Thread satish (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] satish resolved HUDI-1507. -- Resolution: Fixed > Hive sync having issues w/ Clustering > - > >

[jira] [Comment Edited] (HUDI-1509) Major performance degradation due to rewriting records with default values

2021-01-06 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259323#comment-17259323 ] Vinoth Chandar edited comment on HUDI-1509 at 1/6/21, 7:55 PM: --- I timed the

[jira] [Comment Edited] (HUDI-1509) Major performance degradation due to rewriting records with default values

2021-01-06 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259323#comment-17259323 ] Vinoth Chandar edited comment on HUDI-1509 at 1/6/21, 7:54 PM: --- I timed the

[jira] [Assigned] (HUDI-1479) Replace FSUtils.getAllPartitionPaths() with HoodieTableMetadata#getAllPartitionPaths()

2021-01-06 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-1479: Assignee: Udit Mehrotra (was: Vinoth Chandar) > Replace FSUtils.getAllPartitionPaths()

[jira] [Updated] (HUDI-1512) Fix hudi-spark2 unit tests failure with Spark 3.0.0

2021-01-06 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1512: - Labels: pull-request-available (was: ) > Fix hudi-spark2 unit tests failure with Spark 3.0.0 >

[GitHub] [hudi] zhedoubushishi opened a new pull request #2412: [HUDI-1512] Fix spark 2 unit tests failure with Spark 3

2021-01-06 Thread GitBox
zhedoubushishi opened a new pull request #2412: URL: https://github.com/apache/hudi/pull/2412 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of

[GitHub] [hudi] vinothchandar commented on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
vinothchandar commented on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755529282 This points to a deeper discussion. Right now, hudi-common is depended upon by `hudi-hadoop-mr` which has all the Hive/InputFormat code. Should that also be redone as a

[jira] [Comment Edited] (HUDI-1509) Major performance degradation due to rewriting records with default values

2021-01-06 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259988#comment-17259988 ] Nishith Agarwal edited comment on HUDI-1509 at 1/6/21, 7:01 PM:

[GitHub] [hudi] vinothchandar commented on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
vinothchandar commented on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755524721 >Engine can not be tied to client/writing, but can engine become a standalone module? Potentially. We can consider this again, when we think about the additional

[jira] [Updated] (HUDI-1512) Fix hudi-spark2 unit tests failure with Spark 3.0.0

2021-01-06 Thread Wenning Ding (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenning Ding updated HUDI-1512: --- Description: This bug is introduced by https://github.com/apache/hudi/pull/2328.   hudi-spark2 unit

[jira] [Commented] (HUDI-1509) Major performance degradation due to rewriting records with default values

2021-01-06 Thread Nishith Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259988#comment-17259988 ] Nishith Agarwal commented on HUDI-1509: --- [~Pratyaksh] [~uditme] Since you guys implemented/reviewed

[jira] [Updated] (HUDI-1512) Fix hudi-spark2 unit tests failure with Spark 3.0.0

2021-01-06 Thread Wenning Ding (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenning Ding updated HUDI-1512: --- Description: hudi-spark2 unit tests failed when running with Spark 3.0.0: {code:java} mvn clean

[jira] [Assigned] (HUDI-1512) Fix hudi-spark2 unit tests failure with Spark 3.0.0

2021-01-06 Thread Wenning Ding (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenning Ding reassigned HUDI-1512: -- Assignee: Wenning Ding > Fix hudi-spark2 unit tests failure with Spark 3.0.0 >

[GitHub] [hudi] SureshK-T2S commented on issue #2406: [SUPPORT] Deltastreamer - Property hoodie.datasource.write.partitionpath.field not found

2021-01-06 Thread GitBox
SureshK-T2S commented on issue #2406: URL: https://github.com/apache/hudi/issues/2406#issuecomment-755441343 Thanks for your response @bvaradar. Entering the parameters in that way did the trick for this particular issue. ``` spark-submit --class

[GitHub] [hudi] yanghua commented on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
yanghua commented on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755391853 > @yanghua Did mull this a lot, along similar lines. I was thinking about Engine as a general construct that provides parallel execution, rather than being tied to the

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…

2021-01-06 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2334: URL: https://github.com/apache/hudi/pull/2334#discussion_r552744172 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java ## @@ -93,18 +93,14 @@ public

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2334: [HUDI-1453] Throw Exception when input data schema is not equal to th…

2021-01-06 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2334: URL: https://github.com/apache/hudi/pull/2334#discussion_r552740514 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java ## @@ -53,47 +52,70 @@ private static

[GitHub] [hudi] vinothchandar commented on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
vinothchandar commented on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755372065 @yanghua Did mull this a lot, along similar lines. I was thinking about Engine as a general construct that provides parallel execution, rather than being tied to the

[GitHub] [hudi] vinothchandar commented on pull request #2411: [MINOR] Rename unit test package of hudi-spark3 from scala to java

2021-01-06 Thread GitBox
vinothchandar commented on pull request #2411: URL: https://github.com/apache/hudi/pull/2411#issuecomment-755367224 cc @zhedoubushishi @umehrot2 as FYI This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] yanghua commented on a change in pull request #2260: [HUDI-1381] Schedule compaction based on time elapsed

2021-01-06 Thread GitBox
yanghua commented on a change in pull request #2260: URL: https://github.com/apache/hudi/pull/2260#discussion_r552706881 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java ## @@ -60,17

[hudi] branch master updated: [MINOR] Rename unit test package of hudi-spark3 from scala to java (#2411)

2021-01-06 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new b593f10 [MINOR] Rename unit test package of

[GitHub] [hudi] yanghua merged pull request #2411: [MINOR] Rename unit test package of hudi-spark3 from scala to java

2021-01-06 Thread GitBox
yanghua merged pull request #2411: URL: https://github.com/apache/hudi/pull/2411 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [hudi] yui2010 commented on a change in pull request #2378: [HUDI-1491] Support partition pruning for MOR snapshot query

2021-01-06 Thread GitBox
yui2010 commented on a change in pull request #2378: URL: https://github.com/apache/hudi/pull/2378#discussion_r552497329 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/MergeOnReadSnapshotRelation.scala ## @@ -108,7 +111,7 @@ class

[jira] [Commented] (HUDI-1497) Timeout Exception during getFileStatus()

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259759#comment-17259759 ] Steve Loughran commented on HUDI-1497: -- Will happen if you have too few connections in the http pool.

[GitHub] [hudi] codecov-io edited a comment on pull request #2260: [HUDI-1381] Schedule compaction based on time elapsed

2021-01-06 Thread GitBox
codecov-io edited a comment on pull request #2260: URL: https://github.com/apache/hudi/pull/2260#issuecomment-729530724 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] garyli1019 commented on a change in pull request #2375: [HUDI-1332] Introduce FlinkHoodieBloomIndex to hudi-flink-client

2021-01-06 Thread GitBox
garyli1019 commented on a change in pull request #2375: URL: https://github.com/apache/hudi/pull/2375#discussion_r552648364 ## File path: hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/index/bloom/TestFlinkHoodieBloomIndex.java ## @@ -0,0 +1,444 @@ +/* + *

[GitHub] [hudi] codecov-io edited a comment on pull request #2411: [MINOR] Rename unit test package of hudi-spark3 from scala to java

2021-01-06 Thread GitBox
codecov-io edited a comment on pull request #2411: URL: https://github.com/apache/hudi/pull/2411#issuecomment-755280426 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] Karl-WangSK commented on a change in pull request #2260: [HUDI-1381] Schedule compaction based on time elapsed

2021-01-06 Thread GitBox
Karl-WangSK commented on a change in pull request #2260: URL: https://github.com/apache/hudi/pull/2260#discussion_r552590421 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java ## @@ -60,17

[GitHub] [hudi] Karl-WangSK commented on a change in pull request #2260: [HUDI-1381] Schedule compaction based on time elapsed

2021-01-06 Thread GitBox
Karl-WangSK commented on a change in pull request #2260: URL: https://github.com/apache/hudi/pull/2260#discussion_r552590234 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java ## @@ -60,17

[GitHub] [hudi] nsivabalan merged pull request #2407: [HUDI-1507] Change timeline utils to support reading replacecommit

2021-01-06 Thread GitBox
nsivabalan merged pull request #2407: URL: https://github.com/apache/hudi/pull/2407 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[hudi] branch master updated (da2919a -> 2c4868e)

2021-01-06 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from da2919a [HUDI-1383] Fixing sorting of partition vals for hive sync computation (#2402) add 2c4868e

[GitHub] [hudi] nsivabalan merged pull request #2402: [HUDI-1383] Fixing sorting of partition vals for hive sync computation

2021-01-06 Thread GitBox
nsivabalan merged pull request #2402: URL: https://github.com/apache/hudi/pull/2402 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[hudi] branch master updated: [HUDI-1383] Fixing sorting of partition vals for hive sync computation (#2402)

2021-01-06 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new da2919a [HUDI-1383] Fixing sorting of

[GitHub] [hudi] codecov-io commented on pull request #2411: [MINOR] Rename unit test package of hudi-spark3 from scala to java

2021-01-06 Thread GitBox
codecov-io commented on pull request #2411: URL: https://github.com/apache/hudi/pull/2411#issuecomment-755280426 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2411?src=pr=h1) Report > Merging [#2411](https://codecov.io/gh/apache/hudi/pull/2411?src=pr=desc) (f6dbaf7) into

[GitHub] [hudi] yanghua commented on a change in pull request #2260: [HUDI-1381] Schedule compaction based on time elapsed

2021-01-06 Thread GitBox
yanghua commented on a change in pull request #2260: URL: https://github.com/apache/hudi/pull/2260#discussion_r552547254 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/compact/SparkScheduleCompactionActionExecutor.java ## @@ -60,17

[GitHub] [hudi] yanghua commented on pull request #2410: [HUDI-1510] Move HoodieEngineContext and its dependencies to hudi-common

2021-01-06 Thread GitBox
yanghua commented on pull request #2410: URL: https://github.com/apache/hudi/pull/2410#issuecomment-755261406 @vinothchandar I have a concern about this operation. The common module seems not a good place to hold the context of the client engine. It will break the abstraction. The write

[jira] [Closed] (HUDI-1506) Fix wrong exception thrown in HoodieAvroUtils

2021-01-06 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1506. -- Resolution: Fixed Fixed via master branch: 47c5e518a756df815287502a50da9d73d28fc662 > Fix wrong exception

[jira] [Updated] (HUDI-1506) Fix wrong exception thrown in HoodieAvroUtils

2021-01-06 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-1506: --- Fix Version/s: 0.7.0 > Fix wrong exception thrown in HoodieAvroUtils >

[hudi] branch master updated: [HUDI-1506] Fix wrong exception thrown in HoodieAvroUtils (#2405)

2021-01-06 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 47c5e51 [HUDI-1506] Fix wrong exception thrown

[GitHub] [hudi] yanghua merged pull request #2405: [HUDI-1506] Fix wrong exception thrown in HoodieAvroUtils

2021-01-06 Thread GitBox
yanghua merged pull request #2405: URL: https://github.com/apache/hudi/pull/2405 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

  1   2   >