[jira] [Commented] (HUDI-2212) Missing PrimaryKey In Hoodie Properties For CTAS Table
[ https://issues.apache.org/jira/browse/HUDI-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385964#comment-17385964 ] ASF GitHub Bot commented on HUDI-2212: -- hudi-bot edited a comment on pull request #3332: URL: https://github.com/apache/hudi/pull/3332#issuecomment-885403205 ## CI report: * dd84fd3d2f92b35ab104cdb66e7c93f0eaf63a1e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1117) * 83ceb6570a59c91fd26c43fa861d58389064e0d9 UNKNOWN * a26e04ef4bebc35737375563c6e3e6bcf0ac3791 UNKNOWN * 12b69898c69b02eecef6d501021037efcf698bf5 UNKNOWN * b2a6f9e46802725fa50e59a2214861ca9a4915c6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Missing PrimaryKey In Hoodie Properties For CTAS Table > -- > > Key: HUDI-2212 > URL: https://issues.apache.org/jira/browse/HUDI-2212 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Critical > Labels: pull-request-available > > The table created by CTAS has missed the record fields in the > hoodie.properties which will lead the crash for merge & update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3332: [HUDI-2212] Missing PrimaryKey In Hoodie Properties For CTAS Table
hudi-bot edited a comment on pull request #3332: URL: https://github.com/apache/hudi/pull/3332#issuecomment-885403205 ## CI report: * dd84fd3d2f92b35ab104cdb66e7c93f0eaf63a1e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1117) * 83ceb6570a59c91fd26c43fa861d58389064e0d9 UNKNOWN * a26e04ef4bebc35737375563c6e3e6bcf0ac3791 UNKNOWN * 12b69898c69b02eecef6d501021037efcf698bf5 UNKNOWN * b2a6f9e46802725fa50e59a2214861ca9a4915c6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2212) Missing PrimaryKey In Hoodie Properties For CTAS Table
[ https://issues.apache.org/jira/browse/HUDI-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385960#comment-17385960 ] ASF GitHub Bot commented on HUDI-2212: -- hudi-bot edited a comment on pull request #3332: URL: https://github.com/apache/hudi/pull/3332#issuecomment-885403205 ## CI report: * dd84fd3d2f92b35ab104cdb66e7c93f0eaf63a1e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1117) * 83ceb6570a59c91fd26c43fa861d58389064e0d9 UNKNOWN * a26e04ef4bebc35737375563c6e3e6bcf0ac3791 UNKNOWN * 12b69898c69b02eecef6d501021037efcf698bf5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Missing PrimaryKey In Hoodie Properties For CTAS Table > -- > > Key: HUDI-2212 > URL: https://issues.apache.org/jira/browse/HUDI-2212 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Critical > Labels: pull-request-available > > The table created by CTAS has missed the record fields in the > hoodie.properties which will lead the crash for merge & update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3332: [HUDI-2212] Missing PrimaryKey In Hoodie Properties For CTAS Table
hudi-bot edited a comment on pull request #3332: URL: https://github.com/apache/hudi/pull/3332#issuecomment-885403205 ## CI report: * dd84fd3d2f92b35ab104cdb66e7c93f0eaf63a1e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1117) * 83ceb6570a59c91fd26c43fa861d58389064e0d9 UNKNOWN * a26e04ef4bebc35737375563c6e3e6bcf0ac3791 UNKNOWN * 12b69898c69b02eecef6d501021037efcf698bf5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385957#comment-17385957 ] ASF GitHub Bot commented on HUDI-1771: -- swuferhong commented on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-885414491 Hi, @vinothchandar , can you review this PR while your have time? Thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Propagate CDC format for hoodie > --- > > Key: HUDI-1771 > URL: https://issues.apache.org/jira/browse/HUDI-1771 > Project: Apache Hudi > Issue Type: New Feature > Components: Flink Integration >Reporter: Danny Chen >Assignee: Zheng yunhong >Priority: Major > Labels: pull-request-available, sev:normal > Fix For: 0.9.0 > > > Like what we discussed in the dev mailing list: > https://lists.apache.org/thread.html/r31b2d1404e4e043a5f875b78105ba6f9a801e78f265ad91242ad5eb2%40%3Cdev.hudi.apache.org%3E > Keep the change flags make new use cases possible: using HUDI as the unified > storage format for DWD and DWS layer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] swuferhong commented on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie
swuferhong commented on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-885414491 Hi, @vinothchandar , can you review this PR while your have time? Thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2101) support z-order for hudi
[ https://issues.apache.org/jira/browse/HUDI-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385955#comment-17385955 ] ASF GitHub Bot commented on HUDI-2101: -- hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * b474498e6de899ae7a14e17bcf4205402c713824 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1116) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > support z-order for hudi > > > Key: HUDI-2101 > URL: https://issues.apache.org/jira/browse/HUDI-2101 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: tao meng >Assignee: tao meng >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > support z-order for hudi to optimze the query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3330: [HUDI-2101]support z-order for hudi
hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * b474498e6de899ae7a14e17bcf4205402c713824 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1116) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2212) Missing PrimaryKey In Hoodie Properties For CTAS Table
[ https://issues.apache.org/jira/browse/HUDI-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385952#comment-17385952 ] ASF GitHub Bot commented on HUDI-2212: -- pengzhiwei2018 commented on pull request #3332: URL: https://github.com/apache/hudi/pull/3332#issuecomment-885411852 I will land it after the Ci has pass to avoid block others. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Missing PrimaryKey In Hoodie Properties For CTAS Table > -- > > Key: HUDI-2212 > URL: https://issues.apache.org/jira/browse/HUDI-2212 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Critical > Labels: pull-request-available > > The table created by CTAS has missed the record fields in the > hoodie.properties which will lead the crash for merge & update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385954#comment-17385954 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883052706 ## CI report: * 7b789d796e585424651ffbbaf9a176bf6986c26b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1115) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virutal keys support for COW all operations > --- > > Key: HUDI-2176 > URL: https://issues.apache.org/jira/browse/HUDI-2176 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virutal keys support for COW all operations > (merge handle) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table
hudi-bot edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883052706 ## CI report: * 7b789d796e585424651ffbbaf9a176bf6986c26b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1115) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] pengzhiwei2018 commented on pull request #3332: [HUDI-2212] Missing PrimaryKey In Hoodie Properties For CTAS Table
pengzhiwei2018 commented on pull request #3332: URL: https://github.com/apache/hudi/pull/3332#issuecomment-885411852 I will land it after the Ci has pass to avoid block others. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2212) Missing PrimaryKey In Hoodie Properties For CTAS Table
[ https://issues.apache.org/jira/browse/HUDI-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385946#comment-17385946 ] ASF GitHub Bot commented on HUDI-2212: -- hudi-bot edited a comment on pull request #3332: URL: https://github.com/apache/hudi/pull/3332#issuecomment-885403205 ## CI report: * dd84fd3d2f92b35ab104cdb66e7c93f0eaf63a1e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1117) * 83ceb6570a59c91fd26c43fa861d58389064e0d9 UNKNOWN * a26e04ef4bebc35737375563c6e3e6bcf0ac3791 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Missing PrimaryKey In Hoodie Properties For CTAS Table > -- > > Key: HUDI-2212 > URL: https://issues.apache.org/jira/browse/HUDI-2212 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Critical > Labels: pull-request-available > > The table created by CTAS has missed the record fields in the > hoodie.properties which will lead the crash for merge & update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3332: [HUDI-2212] Missing PrimaryKey In Hoodie Properties For CTAS Table
hudi-bot edited a comment on pull request #3332: URL: https://github.com/apache/hudi/pull/3332#issuecomment-885403205 ## CI report: * dd84fd3d2f92b35ab104cdb66e7c93f0eaf63a1e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1117) * 83ceb6570a59c91fd26c43fa861d58389064e0d9 UNKNOWN * a26e04ef4bebc35737375563c6e3e6bcf0ac3791 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2212) Missing PrimaryKey In Hoodie Properties For CTAS Table
[ https://issues.apache.org/jira/browse/HUDI-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385944#comment-17385944 ] ASF GitHub Bot commented on HUDI-2212: -- hudi-bot edited a comment on pull request #3332: URL: https://github.com/apache/hudi/pull/3332#issuecomment-885403205 ## CI report: * dd84fd3d2f92b35ab104cdb66e7c93f0eaf63a1e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1117) * 83ceb6570a59c91fd26c43fa861d58389064e0d9 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Missing PrimaryKey In Hoodie Properties For CTAS Table > -- > > Key: HUDI-2212 > URL: https://issues.apache.org/jira/browse/HUDI-2212 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Critical > Labels: pull-request-available > > The table created by CTAS has missed the record fields in the > hoodie.properties which will lead the crash for merge & update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3332: [HUDI-2212] Missing PrimaryKey In Hoodie Properties For CTAS Table
hudi-bot edited a comment on pull request #3332: URL: https://github.com/apache/hudi/pull/3332#issuecomment-885403205 ## CI report: * dd84fd3d2f92b35ab104cdb66e7c93f0eaf63a1e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1117) * 83ceb6570a59c91fd26c43fa861d58389064e0d9 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2212) Missing PrimaryKey In Hoodie Properties For CTAS Table
[ https://issues.apache.org/jira/browse/HUDI-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385943#comment-17385943 ] ASF GitHub Bot commented on HUDI-2212: -- hudi-bot edited a comment on pull request #3332: URL: https://github.com/apache/hudi/pull/3332#issuecomment-885403205 ## CI report: * dd84fd3d2f92b35ab104cdb66e7c93f0eaf63a1e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1117) * 83ceb6570a59c91fd26c43fa861d58389064e0d9 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Missing PrimaryKey In Hoodie Properties For CTAS Table > -- > > Key: HUDI-2212 > URL: https://issues.apache.org/jira/browse/HUDI-2212 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Critical > Labels: pull-request-available > > The table created by CTAS has missed the record fields in the > hoodie.properties which will lead the crash for merge & update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3332: [HUDI-2212] Missing PrimaryKey In Hoodie Properties For CTAS Table
hudi-bot edited a comment on pull request #3332: URL: https://github.com/apache/hudi/pull/3332#issuecomment-885403205 ## CI report: * dd84fd3d2f92b35ab104cdb66e7c93f0eaf63a1e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1117) * 83ceb6570a59c91fd26c43fa861d58389064e0d9 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2212) Missing PrimaryKey In Hoodie Properties For CTAS Table
[ https://issues.apache.org/jira/browse/HUDI-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385942#comment-17385942 ] ASF GitHub Bot commented on HUDI-2212: -- hudi-bot commented on pull request #3332: URL: https://github.com/apache/hudi/pull/3332#issuecomment-885403205 ## CI report: * dd84fd3d2f92b35ab104cdb66e7c93f0eaf63a1e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Missing PrimaryKey In Hoodie Properties For CTAS Table > -- > > Key: HUDI-2212 > URL: https://issues.apache.org/jira/browse/HUDI-2212 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Critical > Labels: pull-request-available > > The table created by CTAS has missed the record fields in the > hoodie.properties which will lead the crash for merge & update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot commented on pull request #3332: [HUDI-2212] Missing PrimaryKey In Hoodie Properties For CTAS Table
hudi-bot commented on pull request #3332: URL: https://github.com/apache/hudi/pull/3332#issuecomment-885403205 ## CI report: * dd84fd3d2f92b35ab104cdb66e7c93f0eaf63a1e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-2212) Missing PrimaryKey In Hoodie Properties For CTAS Table
[ https://issues.apache.org/jira/browse/HUDI-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2212: - Labels: pull-request-available (was: ) > Missing PrimaryKey In Hoodie Properties For CTAS Table > -- > > Key: HUDI-2212 > URL: https://issues.apache.org/jira/browse/HUDI-2212 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Critical > Labels: pull-request-available > > The table created by CTAS has missed the record fields in the > hoodie.properties which will lead the crash for merge & update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2212) Missing PrimaryKey In Hoodie Properties For CTAS Table
[ https://issues.apache.org/jira/browse/HUDI-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385940#comment-17385940 ] ASF GitHub Bot commented on HUDI-2212: -- pengzhiwei2018 opened a new pull request #3332: URL: https://github.com/apache/hudi/pull/3332 ## What is the purpose of the pull request The table created by CTAS has missed the record fields in the hoodie.properties which will lead the crash for merge & update. This PR try to fix this bug. Options ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Missing PrimaryKey In Hoodie Properties For CTAS Table > -- > > Key: HUDI-2212 > URL: https://issues.apache.org/jira/browse/HUDI-2212 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Critical > > The table created by CTAS has missed the record fields in the > hoodie.properties which will lead the crash for merge & update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] pengzhiwei2018 opened a new pull request #3332: [HUDI-2212] Missing PrimaryKey In Hoodie Properties For CTAS Table
pengzhiwei2018 opened a new pull request #3332: URL: https://github.com/apache/hudi/pull/3332 ## What is the purpose of the pull request The table created by CTAS has missed the record fields in the hoodie.properties which will lead the crash for merge & update. This PR try to fix this bug. Options ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2090) when hudi metadata is enabled, use different user to query table, the query will failed
[ https://issues.apache.org/jira/browse/HUDI-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385937#comment-17385937 ] ASF GitHub Bot commented on HUDI-2090: -- hudi-bot edited a comment on pull request #3329: URL: https://github.com/apache/hudi/pull/3329#issuecomment-885330993 ## CI report: * a8c0846435a59cb7422c8962db7f121d72ced322 UNKNOWN * 9f3807615378aaeb6bd901b240346271f5346c28 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1114) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > when hudi metadata is enabled, use different user to query table, the query > will failed > - > > Key: HUDI-2090 > URL: https://issues.apache.org/jira/browse/HUDI-2090 > Project: Apache Hudi > Issue Type: Bug > Components: Common Core >Affects Versions: 0.8.0 >Reporter: tao meng >Assignee: tao meng >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > when hudi metadata is enabled, use different user to query table, the query > will failed. > > The user permissions of the temporary directory generated by DiskBasedMap are > incorrect. This directory only has permissions for the user of current > operation, and other users have no permissions to access it, which leads to > this problem > test step： > step1: create hudi table with metadata enabled. > step1： create two user（omm，user2） > step2: > f1) use omm to query hudi table > DiskBasedMap will generate view_map with permissions drwx--. > 2) then user user2 to query hudi table > now user2 has no right to access view_map which created by omm, the > exception will throws: > org.apache.hudi.exception.HoodieIOException: IOException when creating > ExternalSplillableMap at /tmp/view_map > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3329: [HUDI-2090] Ensure Disk Maps create a subfolder with appropriate prefixes and cleans them up on close
hudi-bot edited a comment on pull request #3329: URL: https://github.com/apache/hudi/pull/3329#issuecomment-885330993 ## CI report: * a8c0846435a59cb7422c8962db7f121d72ced322 UNKNOWN * 9f3807615378aaeb6bd901b240346271f5346c28 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1114) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2101) support z-order for hudi
[ https://issues.apache.org/jira/browse/HUDI-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385935#comment-17385935 ] ASF GitHub Bot commented on HUDI-2101: -- hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 93897c10f0635eb163c9c98894092b034e3fcb14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1112) * b474498e6de899ae7a14e17bcf4205402c713824 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1116) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > support z-order for hudi > > > Key: HUDI-2101 > URL: https://issues.apache.org/jira/browse/HUDI-2101 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: tao meng >Assignee: tao meng >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > support z-order for hudi to optimze the query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3330: [HUDI-2101]support z-order for hudi
hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 93897c10f0635eb163c9c98894092b034e3fcb14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1112) * b474498e6de899ae7a14e17bcf4205402c713824 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1116) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2101) support z-order for hudi
[ https://issues.apache.org/jira/browse/HUDI-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385933#comment-17385933 ] ASF GitHub Bot commented on HUDI-2101: -- hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 93897c10f0635eb163c9c98894092b034e3fcb14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1112) * b474498e6de899ae7a14e17bcf4205402c713824 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > support z-order for hudi > > > Key: HUDI-2101 > URL: https://issues.apache.org/jira/browse/HUDI-2101 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: tao meng >Assignee: tao meng >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > support z-order for hudi to optimze the query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table
hudi-bot edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883052706 ## CI report: * 4970e910283b9ed15662070c8f45eb6a4f769f3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1107) * 7b789d796e585424651ffbbaf9a176bf6986c26b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1115) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3330: [HUDI-2101]support z-order for hudi
hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 93897c10f0635eb163c9c98894092b034e3fcb14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1112) * b474498e6de899ae7a14e17bcf4205402c713824 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385932#comment-17385932 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883052706 ## CI report: * 4970e910283b9ed15662070c8f45eb6a4f769f3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1107) * 7b789d796e585424651ffbbaf9a176bf6986c26b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1115) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virutal keys support for COW all operations > --- > > Key: HUDI-2176 > URL: https://issues.apache.org/jira/browse/HUDI-2176 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virutal keys support for COW all operations > (merge handle) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385931#comment-17385931 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883052706 ## CI report: * 4970e910283b9ed15662070c8f45eb6a4f769f3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1107) * 7b789d796e585424651ffbbaf9a176bf6986c26b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virutal keys support for COW all operations > --- > > Key: HUDI-2176 > URL: https://issues.apache.org/jira/browse/HUDI-2176 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virutal keys support for COW all operations > (merge handle) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table
hudi-bot edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883052706 ## CI report: * 4970e910283b9ed15662070c8f45eb6a4f769f3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1107) * 7b789d796e585424651ffbbaf9a176bf6986c26b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2101) support z-order for hudi
[ https://issues.apache.org/jira/browse/HUDI-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385927#comment-17385927 ] ASF GitHub Bot commented on HUDI-2101: -- hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 93897c10f0635eb163c9c98894092b034e3fcb14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1112) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > support z-order for hudi > > > Key: HUDI-2101 > URL: https://issues.apache.org/jira/browse/HUDI-2101 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: tao meng >Assignee: tao meng >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > support z-order for hudi to optimze the query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3330: [HUDI-2101][WIP]support z-order for hudi
hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 93897c10f0635eb163c9c98894092b034e3fcb14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1112) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2090) when hudi metadata is enabled, use different user to query table, the query will failed
[ https://issues.apache.org/jira/browse/HUDI-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385925#comment-17385925 ] ASF GitHub Bot commented on HUDI-2090: -- hudi-bot edited a comment on pull request #3329: URL: https://github.com/apache/hudi/pull/3329#issuecomment-885330993 ## CI report: * 1b46f976d18083ee2ad7a64c9eaa9a4f19fcb666 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1108) * a8c0846435a59cb7422c8962db7f121d72ced322 UNKNOWN * 9f3807615378aaeb6bd901b240346271f5346c28 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1114) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > when hudi metadata is enabled, use different user to query table, the query > will failed > - > > Key: HUDI-2090 > URL: https://issues.apache.org/jira/browse/HUDI-2090 > Project: Apache Hudi > Issue Type: Bug > Components: Common Core >Affects Versions: 0.8.0 >Reporter: tao meng >Assignee: tao meng >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > when hudi metadata is enabled, use different user to query table, the query > will failed. > > The user permissions of the temporary directory generated by DiskBasedMap are > incorrect. This directory only has permissions for the user of current > operation, and other users have no permissions to access it, which leads to > this problem > test step： > step1: create hudi table with metadata enabled. > step1： create two user（omm，user2） > step2: > f1) use omm to query hudi table > DiskBasedMap will generate view_map with permissions drwx--. > 2) then user user2 to query hudi table > now user2 has no right to access view_map which created by omm, the > exception will throws: > org.apache.hudi.exception.HoodieIOException: IOException when creating > ExternalSplillableMap at /tmp/view_map > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3329: [HUDI-2090] Ensure Disk Maps create a subfolder with appropriate prefixes and cleans them up on close
hudi-bot edited a comment on pull request #3329: URL: https://github.com/apache/hudi/pull/3329#issuecomment-885330993 ## CI report: * 1b46f976d18083ee2ad7a64c9eaa9a4f19fcb666 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1108) * a8c0846435a59cb7422c8962db7f121d72ced322 UNKNOWN * 9f3807615378aaeb6bd901b240346271f5346c28 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1114) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2090) when hudi metadata is enabled, use different user to query table, the query will failed
[ https://issues.apache.org/jira/browse/HUDI-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385924#comment-17385924 ] ASF GitHub Bot commented on HUDI-2090: -- hudi-bot edited a comment on pull request #3329: URL: https://github.com/apache/hudi/pull/3329#issuecomment-885330993 ## CI report: * 1b46f976d18083ee2ad7a64c9eaa9a4f19fcb666 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1108) * a8c0846435a59cb7422c8962db7f121d72ced322 UNKNOWN * 9f3807615378aaeb6bd901b240346271f5346c28 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > when hudi metadata is enabled, use different user to query table, the query > will failed > - > > Key: HUDI-2090 > URL: https://issues.apache.org/jira/browse/HUDI-2090 > Project: Apache Hudi > Issue Type: Bug > Components: Common Core >Affects Versions: 0.8.0 >Reporter: tao meng >Assignee: tao meng >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > when hudi metadata is enabled, use different user to query table, the query > will failed. > > The user permissions of the temporary directory generated by DiskBasedMap are > incorrect. This directory only has permissions for the user of current > operation, and other users have no permissions to access it, which leads to > this problem > test step： > step1: create hudi table with metadata enabled. > step1： create two user（omm，user2） > step2: > f1) use omm to query hudi table > DiskBasedMap will generate view_map with permissions drwx--. > 2) then user user2 to query hudi table > now user2 has no right to access view_map which created by omm, the > exception will throws: > org.apache.hudi.exception.HoodieIOException: IOException when creating > ExternalSplillableMap at /tmp/view_map > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3329: [HUDI-2090] Ensure Disk Maps create a subfolder with appropriate prefixes and cleans them up on close
hudi-bot edited a comment on pull request #3329: URL: https://github.com/apache/hudi/pull/3329#issuecomment-885330993 ## CI report: * 1b46f976d18083ee2ad7a64c9eaa9a4f19fcb666 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1108) * a8c0846435a59cb7422c8962db7f121d72ced322 UNKNOWN * 9f3807615378aaeb6bd901b240346271f5346c28 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2090) when hudi metadata is enabled, use different user to query table, the query will failed
[ https://issues.apache.org/jira/browse/HUDI-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385923#comment-17385923 ] ASF GitHub Bot commented on HUDI-2090: -- hudi-bot edited a comment on pull request #3329: URL: https://github.com/apache/hudi/pull/3329#issuecomment-885330993 ## CI report: * 1b46f976d18083ee2ad7a64c9eaa9a4f19fcb666 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1108) * a8c0846435a59cb7422c8962db7f121d72ced322 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > when hudi metadata is enabled, use different user to query table, the query > will failed > - > > Key: HUDI-2090 > URL: https://issues.apache.org/jira/browse/HUDI-2090 > Project: Apache Hudi > Issue Type: Bug > Components: Common Core >Affects Versions: 0.8.0 >Reporter: tao meng >Assignee: tao meng >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > when hudi metadata is enabled, use different user to query table, the query > will failed. > > The user permissions of the temporary directory generated by DiskBasedMap are > incorrect. This directory only has permissions for the user of current > operation, and other users have no permissions to access it, which leads to > this problem > test step： > step1: create hudi table with metadata enabled. > step1： create two user（omm，user2） > step2: > f1) use omm to query hudi table > DiskBasedMap will generate view_map with permissions drwx--. > 2) then user user2 to query hudi table > now user2 has no right to access view_map which created by omm, the > exception will throws: > org.apache.hudi.exception.HoodieIOException: IOException when creating > ExternalSplillableMap at /tmp/view_map > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3329: [HUDI-2090] Ensure Disk Maps create a subfolder with appropriate prefixes and cleans them up on close
hudi-bot edited a comment on pull request #3329: URL: https://github.com/apache/hudi/pull/3329#issuecomment-885330993 ## CI report: * 1b46f976d18083ee2ad7a64c9eaa9a4f19fcb666 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1108) * a8c0846435a59cb7422c8962db7f121d72ced322 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HUDI-2212) Missing PrimaryKey In Hoodie Properties For CTAS Table
[ https://issues.apache.org/jira/browse/HUDI-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengzhiwei reassigned HUDI-2212: Assignee: pengzhiwei > Missing PrimaryKey In Hoodie Properties For CTAS Table > -- > > Key: HUDI-2212 > URL: https://issues.apache.org/jira/browse/HUDI-2212 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Major > > The table created by CTAS has missed the record fields in the > hoodie.properties which will lead the crash for merge & update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-2212) Missing PrimaryKey In Hoodie Properties For CTAS Table
[ https://issues.apache.org/jira/browse/HUDI-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengzhiwei updated HUDI-2212: - Priority: Critical (was: Major) > Missing PrimaryKey In Hoodie Properties For CTAS Table > -- > > Key: HUDI-2212 > URL: https://issues.apache.org/jira/browse/HUDI-2212 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: pengzhiwei >Assignee: pengzhiwei >Priority: Critical > > The table created by CTAS has missed the record fields in the > hoodie.properties which will lead the crash for merge & update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HUDI-2212) Missing PrimaryKey In Hoodie Properties For CTAS Table
pengzhiwei created HUDI-2212: Summary: Missing PrimaryKey In Hoodie Properties For CTAS Table Key: HUDI-2212 URL: https://issues.apache.org/jira/browse/HUDI-2212 Project: Apache Hudi Issue Type: Sub-task Components: Spark Integration Reporter: pengzhiwei The table created by CTAS has missed the record fields in the hoodie.properties which will lead the crash for merge & update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[hudi] branch master updated (5a2f3d4 -> 6d592c5)
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 5a2f3d4 [HUDI-2139] MergeInto MOR Table May Result InCorrect Result (#3230) add 6d592c5 [HUDI-2211] Fix NullPointerException in TestHoodieConsoleMetrics (#3331) No new revisions were added by this update. Summary of changes: .../src/test/java/org/apache/hudi/metrics/TestHoodieConsoleMetrics.java | 1 + 1 file changed, 1 insertion(+)
[jira] [Updated] (HUDI-2211) Fix NullPointerException in TestHoodieConsoleMetrics
[ https://issues.apache.org/jira/browse/HUDI-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-2211: --- Fix Version/s: 0.9.0 > Fix NullPointerException in TestHoodieConsoleMetrics > > > Key: HUDI-2211 > URL: https://issues.apache.org/jira/browse/HUDI-2211 > Project: Apache Hudi > Issue Type: Improvement > Components: metrics >Reporter: Xuedong Luan >Assignee: Xuedong Luan >Priority: Minor > Labels: pull-request-available > Fix For: 0.9.0 > > > java.lang.NullPointerException: Expected a non-null value. Got > nulljava.lang.NullPointerException: Expected a non-null value. Got null at > org.apache.hudi.common.util.Option.(Option.java:65) at > org.apache.hudi.common.util.Option.of(Option.java:75) at > org.apache.hudi.metrics.Metrics.registerHoodieCommonMetrics(Metrics.java:85) > at org.apache.hudi.metrics.Metrics.reportAndCloseReporter(Metrics.java:63) at > org.apache.hudi.metrics.Metrics.shutdown(Metrics.java:109) at > org.apache.hudi.metrics.TestHoodieConsoleMetrics.stop(TestHoodieConsoleMetrics.java:48) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (HUDI-2211) Fix NullPointerException in TestHoodieConsoleMetrics
[ https://issues.apache.org/jira/browse/HUDI-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-2211. -- Resolution: Done 6d592c5896d033c7f781d4a9eef4a43916c084ed > Fix NullPointerException in TestHoodieConsoleMetrics > > > Key: HUDI-2211 > URL: https://issues.apache.org/jira/browse/HUDI-2211 > Project: Apache Hudi > Issue Type: Improvement > Components: metrics >Reporter: Xuedong Luan >Assignee: Xuedong Luan >Priority: Minor > Labels: pull-request-available > Fix For: 0.9.0 > > > java.lang.NullPointerException: Expected a non-null value. Got > nulljava.lang.NullPointerException: Expected a non-null value. Got null at > org.apache.hudi.common.util.Option.(Option.java:65) at > org.apache.hudi.common.util.Option.of(Option.java:75) at > org.apache.hudi.metrics.Metrics.registerHoodieCommonMetrics(Metrics.java:85) > at org.apache.hudi.metrics.Metrics.reportAndCloseReporter(Metrics.java:63) at > org.apache.hudi.metrics.Metrics.shutdown(Metrics.java:109) at > org.apache.hudi.metrics.TestHoodieConsoleMetrics.stop(TestHoodieConsoleMetrics.java:48) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2211) Fix NullPointerException in TestHoodieConsoleMetrics
[ https://issues.apache.org/jira/browse/HUDI-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385904#comment-17385904 ] ASF GitHub Bot commented on HUDI-2211: -- yanghua merged pull request #3331: URL: https://github.com/apache/hudi/pull/3331 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix NullPointerException in TestHoodieConsoleMetrics > > > Key: HUDI-2211 > URL: https://issues.apache.org/jira/browse/HUDI-2211 > Project: Apache Hudi > Issue Type: Improvement > Components: metrics >Reporter: Xuedong Luan >Assignee: Xuedong Luan >Priority: Minor > Labels: pull-request-available > > java.lang.NullPointerException: Expected a non-null value. Got > nulljava.lang.NullPointerException: Expected a non-null value. Got null at > org.apache.hudi.common.util.Option.(Option.java:65) at > org.apache.hudi.common.util.Option.of(Option.java:75) at > org.apache.hudi.metrics.Metrics.registerHoodieCommonMetrics(Metrics.java:85) > at org.apache.hudi.metrics.Metrics.reportAndCloseReporter(Metrics.java:63) at > org.apache.hudi.metrics.Metrics.shutdown(Metrics.java:109) at > org.apache.hudi.metrics.TestHoodieConsoleMetrics.stop(TestHoodieConsoleMetrics.java:48) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] yanghua merged pull request #3331: [HUDI-2211] Fix NullPointerException in TestHoodieConsoleMetrics
yanghua merged pull request #3331: URL: https://github.com/apache/hudi/pull/3331 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385903#comment-17385903 ] ASF GitHub Bot commented on HUDI-2176: -- nsivabalan commented on a change in pull request #3306: URL: https://github.com/apache/hudi/pull/3306#discussion_r675284515 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/SparkKeyGeneratorInterface.java ## @@ -28,4 +30,6 @@ String getRecordKey(Row row); String getPartitionPath(Row row); + + String getPartitionPath(InternalRow internalRow, StructType structType); Review comment: I followed what we did when we introduced Row to these interfaces. We added default impl only in BuiltInKeyGen. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Virutal keys support for COW all operations > --- > > Key: HUDI-2176 > URL: https://issues.apache.org/jira/browse/HUDI-2176 > Project: Apache Hudi > Issue Type: Improvement > Components: Writer Core >Reporter: sivabalan narayanan >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Virutal keys support for COW all operations > (merge handle) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] nsivabalan commented on a change in pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table
nsivabalan commented on a change in pull request #3306: URL: https://github.com/apache/hudi/pull/3306#discussion_r675284515 ## File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/keygen/SparkKeyGeneratorInterface.java ## @@ -28,4 +30,6 @@ String getRecordKey(Row row); String getPartitionPath(Row row); + + String getPartitionPath(InternalRow internalRow, StructType structType); Review comment: I followed what we did when we introduced Row to these interfaces. We added default impl only in BuiltInKeyGen. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[
https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385896#comment-17385896
]
ASF GitHub Bot commented on HUDI-2176:
--
nsivabalan commented on a change in pull request #3306:
URL: https://github.com/apache/hudi/pull/3306#discussion_r675282410
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/ParquetUtils.java
##
@@ -142,6 +143,43 @@
return hoodieKeys;
}
+ /**
+ * Fetch {@link HoodieKey}s from the given parquet file.
+ *
+ * @param filePath The parquet file path.
+ * @param configuration configuration to build fs object
+ * @return {@link List} of {@link HoodieKey}s fetched from the parquet file
+ */
+ @Override
+ public List fetchRecordKeyPartitionPath(Configuration
configuration, Path filePath, BaseKeyGenerator keyGenerator) {
+List hoodieKeys = new ArrayList<>();
+try {
+ if (!filePath.getFileSystem(configuration).exists(filePath)) {
Review comment:
existing fetchRecordKeyPartitionPath() already does this. I assume you
suggested to fix all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Virutal keys support for COW all operations
> ---
>
> Key: HUDI-2176
> URL: https://issues.apache.org/jira/browse/HUDI-2176
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Writer Core
>Reporter: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Virutal keys support for COW all operations
> (merge handle)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] nsivabalan commented on a change in pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table
nsivabalan commented on a change in pull request #3306:
URL: https://github.com/apache/hudi/pull/3306#discussion_r675282410
##
File path:
hudi-common/src/main/java/org/apache/hudi/common/util/ParquetUtils.java
##
@@ -142,6 +143,43 @@
return hoodieKeys;
}
+ /**
+ * Fetch {@link HoodieKey}s from the given parquet file.
+ *
+ * @param filePath The parquet file path.
+ * @param configuration configuration to build fs object
+ * @return {@link List} of {@link HoodieKey}s fetched from the parquet file
+ */
+ @Override
+ public List fetchRecordKeyPartitionPath(Configuration
configuration, Path filePath, BaseKeyGenerator keyGenerator) {
+List hoodieKeys = new ArrayList<>();
+try {
+ if (!filePath.getFileSystem(configuration).exists(filePath)) {
Review comment:
existing fetchRecordKeyPartitionPath() already does this. I assume you
suggested to fix all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-2211) Fix NullPointerException in TestHoodieConsoleMetrics
[ https://issues.apache.org/jira/browse/HUDI-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385894#comment-17385894 ] ASF GitHub Bot commented on HUDI-2211: -- hudi-bot edited a comment on pull request #3331: URL: https://github.com/apache/hudi/pull/3331#issuecomment-885351519 ## CI report: * fa7d1d55847d21fb5d89451b1785c479ec77554d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1110) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix NullPointerException in TestHoodieConsoleMetrics > > > Key: HUDI-2211 > URL: https://issues.apache.org/jira/browse/HUDI-2211 > Project: Apache Hudi > Issue Type: Improvement > Components: metrics >Reporter: Xuedong Luan >Assignee: Xuedong Luan >Priority: Minor > Labels: pull-request-available > > java.lang.NullPointerException: Expected a non-null value. Got > nulljava.lang.NullPointerException: Expected a non-null value. Got null at > org.apache.hudi.common.util.Option.(Option.java:65) at > org.apache.hudi.common.util.Option.of(Option.java:75) at > org.apache.hudi.metrics.Metrics.registerHoodieCommonMetrics(Metrics.java:85) > at org.apache.hudi.metrics.Metrics.reportAndCloseReporter(Metrics.java:63) at > org.apache.hudi.metrics.Metrics.shutdown(Metrics.java:109) at > org.apache.hudi.metrics.TestHoodieConsoleMetrics.stop(TestHoodieConsoleMetrics.java:48) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2101) support z-order for hudi
[ https://issues.apache.org/jira/browse/HUDI-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385893#comment-17385893 ] ASF GitHub Bot commented on HUDI-2101: -- hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 5a7e153b3b8cdf2d5922839db356282b01dc8d92 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1109) * 93897c10f0635eb163c9c98894092b034e3fcb14 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1112) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > support z-order for hudi > > > Key: HUDI-2101 > URL: https://issues.apache.org/jira/browse/HUDI-2101 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: tao meng >Assignee: tao meng >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > support z-order for hudi to optimze the query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3331: [HUDI-2211] Fix NullPointerException in TestHoodieConsoleMetrics
hudi-bot edited a comment on pull request #3331: URL: https://github.com/apache/hudi/pull/3331#issuecomment-885351519 ## CI report: * fa7d1d55847d21fb5d89451b1785c479ec77554d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1110) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3330: [HUDI-2101][WIP]support z-order for hudi
hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 5a7e153b3b8cdf2d5922839db356282b01dc8d92 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1109) * 93897c10f0635eb163c9c98894092b034e3fcb14 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1112) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[
https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385885#comment-17385885
]
ASF GitHub Bot commented on HUDI-2176:
--
nsivabalan commented on a change in pull request #3306:
URL: https://github.com/apache/hudi/pull/3306#discussion_r675277185
##
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##
@@ -244,6 +244,22 @@ public static Schema getRecordKeyPartitionPathSchema() {
return recordSchema;
}
+ /**
+ * Fetch schema for record key and partition path.
+ */
+ public static Schema getRecordKeyPartitionPathSchema(Schema fileSchema,
List recordKeyFields, List partitionPathFields) {
Review comment:
looks like we don't have one already. But will fix this method to be
generic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Virutal keys support for COW all operations
> ---
>
> Key: HUDI-2176
> URL: https://issues.apache.org/jira/browse/HUDI-2176
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Writer Core
>Reporter: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Virutal keys support for COW all operations
> (merge handle)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] nsivabalan commented on a change in pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table
nsivabalan commented on a change in pull request #3306:
URL: https://github.com/apache/hudi/pull/3306#discussion_r675277185
##
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##
@@ -244,6 +244,22 @@ public static Schema getRecordKeyPartitionPathSchema() {
return recordSchema;
}
+ /**
+ * Fetch schema for record key and partition path.
+ */
+ public static Schema getRecordKeyPartitionPathSchema(Schema fileSchema,
List recordKeyFields, List partitionPathFields) {
Review comment:
looks like we don't have one already. But will fix this method to be
generic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385879#comment-17385879 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 removed a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-885360664 Is there a partner who can help me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385878#comment-17385878 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-885360664 Is there a partner who can help me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] liujinhui1994 commented on issue #3280: [SUPPORT] Use structedstreaming to consume kafka to write to hudi error
liujinhui1994 commented on issue #3280: URL: https://github.com/apache/hudi/issues/3280#issuecomment-885360852 Is there a partner who can help me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] liujinhui1994 removed a comment on pull request #3312: [HUDI-648][RFC-20] Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
liujinhui1994 removed a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-885360664 Is there a partner who can help me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on pull request #3312: [HUDI-648][RFC-20] Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
liujinhui1994 commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-885360664 Is there a partner who can help me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385874#comment-17385874 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-885357803 OK, let's get started. Your guidance please @lw309637554 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] liujinhui1994 commented on pull request #3312: [HUDI-648][RFC-20] Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
liujinhui1994 commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-885357803 OK, let's get started. Your guidance please @lw309637554 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2164) Build cluster plan and execute this plan at once for HoodieClusteringJob
[ https://issues.apache.org/jira/browse/HUDI-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385871#comment-17385871 ] ASF GitHub Bot commented on HUDI-2164: -- lw309637554 commented on pull request #3259: URL: https://github.com/apache/hudi/pull/3259#issuecomment-885356337 @zhangyue19921010 hi, some minor comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Build cluster plan and execute this plan at once for HoodieClusteringJob > > > Key: HUDI-2164 > URL: https://issues.apache.org/jira/browse/HUDI-2164 > Project: Apache Hudi > Issue Type: Task >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > > For now, Hudi can let users submit a HoodieClusteringJob to build a > clustering plan or execute a clustering plan through --schedule or > --instant-time config. > If users want to trigger a clustering job, he has to > # Submit a HoodieClusteringJob to build a clustering job through --schedule > config > # Copy the created clustering Instant time form Log info. > # Submit the HoodieClusteringJob again to execute this created clustering > plan through --instant-time config. > The pain point is that there are too many steps when trigger a clustering and > need to copy and paste the instant time from log file manually so that we > can't make it automatically. > > I just raise a PR to offer a new config named --mode or -m in short > ||--mode||remarks|| > |execute|Execute a cluster plan at given instant which means --instant-time > is needed here. default value. | > |schedule|Make a clustering plan.| > |*scheduleAndExecute*|Make a cluster plan first and execute that plan > immediately| > Now users can use --mode scheduleAndExecute to Build cluster plan and execute > this plan at once using HoodieClusteringJob. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] lw309637554 commented on pull request #3259: [HUDI-2164] Let users build cluster plan and execute this plan at once using HoodieClusteringJob for async clustering
lw309637554 commented on pull request #3259: URL: https://github.com/apache/hudi/pull/3259#issuecomment-885356337 @zhangyue19921010 hi, some minor comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2164) Build cluster plan and execute this plan at once for HoodieClusteringJob
[
https://issues.apache.org/jira/browse/HUDI-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385870#comment-17385870
]
ASF GitHub Bot commented on HUDI-2164:
--
lw309637554 commented on a change in pull request #3259:
URL: https://github.com/apache/hudi/pull/3259#discussion_r675270701
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##
@@ -449,6 +451,14 @@ static void assertAtleastNDeltaCommits(int minExpected,
String tablePath, FileSy
assertTrue(minExpected <= numDeltaCommits, "Got=" + numDeltaCommits + ",
exp >=" + minExpected);
}
+static void assertAtLeastNCompletedReplaceCommits(int minExpected, String
tablePath, DistributedFileSystem fs) {
+ HoodieTableMetaClient meta =
HoodieTableMetaClient.builder().setConf(fs.getConf()).setLoadActiveTimelineOnLoad(true).setBasePath(tablePath).build();
+ HoodieTimeline timeline =
meta.getActiveTimeline().getCompletedReplaceTimeline();
+ LOG.info("Timeline Instants=" +
meta.getActiveTimeline().getInstants().collect(Collectors.toList()));
Review comment:
Timeline Instants= -> Timeline instants =
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Build cluster plan and execute this plan at once for HoodieClusteringJob
>
>
> Key: HUDI-2164
> URL: https://issues.apache.org/jira/browse/HUDI-2164
> Project: Apache Hudi
> Issue Type: Task
>Reporter: Yue Zhang
>Priority: Major
> Labels: pull-request-available
>
> For now, Hudi can let users submit a HoodieClusteringJob to build a
> clustering plan or execute a clustering plan through --schedule or
> --instant-time config.
> If users want to trigger a clustering job, he has to
> # Submit a HoodieClusteringJob to build a clustering job through --schedule
> config
> # Copy the created clustering Instant time form Log info.
> # Submit the HoodieClusteringJob again to execute this created clustering
> plan through --instant-time config.
> The pain point is that there are too many steps when trigger a clustering and
> need to copy and paste the instant time from log file manually so that we
> can't make it automatically.
>
> I just raise a PR to offer a new config named --mode or -m in short
> ||--mode||remarks||
> |execute|Execute a cluster plan at given instant which means --instant-time
> is needed here. default value. |
> |schedule|Make a clustering plan.|
> |*scheduleAndExecute*|Make a cluster plan first and execute that plan
> immediately|
> Now users can use --mode scheduleAndExecute to Build cluster plan and execute
> this plan at once using HoodieClusteringJob.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-2164) Build cluster plan and execute this plan at once for HoodieClusteringJob
[
https://issues.apache.org/jira/browse/HUDI-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385869#comment-17385869
]
ASF GitHub Bot commented on HUDI-2164:
--
lw309637554 commented on a change in pull request #3259:
URL: https://github.com/apache/hudi/pull/3259#discussion_r675270601
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieClusteringJob.java
##
@@ -164,11 +196,37 @@ private int doCluster(JavaSparkContext jsc) throws
Exception {
private Option doSchedule(JavaSparkContext jsc) throws Exception {
String schemaStr = getSchemaFromLatestInstant();
try (SparkRDDWriteClient client =
UtilHelpers.createHoodieClient(jsc, cfg.basePath, schemaStr, cfg.parallelism,
Option.empty(), props)) {
- if (cfg.clusteringInstantTime != null) {
-client.scheduleClusteringAtInstant(cfg.clusteringInstantTime,
Option.empty());
-return Option.of(cfg.clusteringInstantTime);
+ return doSchedule(client);
+}
+ }
+
+ private Option doSchedule(SparkRDDWriteClient
client) {
+if (cfg.clusteringInstantTime != null) {
+ client.scheduleClusteringAtInstant(cfg.clusteringInstantTime,
Option.empty());
+ return Option.of(cfg.clusteringInstantTime);
+}
+return client.scheduleClustering(Option.empty());
+ }
+
+ public int doScheduleAndCluster(JavaSparkContext jsc) throws Exception {
+LOG.info("Step 1: Do schedule");
+String schemaStr = getSchemaFromLatestInstant();
+try (SparkRDDWriteClient client =
UtilHelpers.createHoodieClient(jsc, cfg.basePath, schemaStr, cfg.parallelism,
Option.empty(), props)) {
+
+ Option instantTime = doSchedule(client);
+ int result = instantTime.isPresent() ? 0 : -1;
+
+ if (result == -1) {
+LOG.info("Couldn't Generate Cluster Plan");
Review comment:
Couldn't Generate Cluster Plan -> Couldn't generate cluster plan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Build cluster plan and execute this plan at once for HoodieClusteringJob
>
>
> Key: HUDI-2164
> URL: https://issues.apache.org/jira/browse/HUDI-2164
> Project: Apache Hudi
> Issue Type: Task
>Reporter: Yue Zhang
>Priority: Major
> Labels: pull-request-available
>
> For now, Hudi can let users submit a HoodieClusteringJob to build a
> clustering plan or execute a clustering plan through --schedule or
> --instant-time config.
> If users want to trigger a clustering job, he has to
> # Submit a HoodieClusteringJob to build a clustering job through --schedule
> config
> # Copy the created clustering Instant time form Log info.
> # Submit the HoodieClusteringJob again to execute this created clustering
> plan through --instant-time config.
> The pain point is that there are too many steps when trigger a clustering and
> need to copy and paste the instant time from log file manually so that we
> can't make it automatically.
>
> I just raise a PR to offer a new config named --mode or -m in short
> ||--mode||remarks||
> |execute|Execute a cluster plan at given instant which means --instant-time
> is needed here. default value. |
> |schedule|Make a clustering plan.|
> |*scheduleAndExecute*|Make a cluster plan first and execute that plan
> immediately|
> Now users can use --mode scheduleAndExecute to Build cluster plan and execute
> this plan at once using HoodieClusteringJob.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] lw309637554 commented on a change in pull request #3259: [HUDI-2164] Let users build cluster plan and execute this plan at once using HoodieClusteringJob for async clustering
lw309637554 commented on a change in pull request #3259:
URL: https://github.com/apache/hudi/pull/3259#discussion_r675270701
##
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##
@@ -449,6 +451,14 @@ static void assertAtleastNDeltaCommits(int minExpected,
String tablePath, FileSy
assertTrue(minExpected <= numDeltaCommits, "Got=" + numDeltaCommits + ",
exp >=" + minExpected);
}
+static void assertAtLeastNCompletedReplaceCommits(int minExpected, String
tablePath, DistributedFileSystem fs) {
+ HoodieTableMetaClient meta =
HoodieTableMetaClient.builder().setConf(fs.getConf()).setLoadActiveTimelineOnLoad(true).setBasePath(tablePath).build();
+ HoodieTimeline timeline =
meta.getActiveTimeline().getCompletedReplaceTimeline();
+ LOG.info("Timeline Instants=" +
meta.getActiveTimeline().getInstants().collect(Collectors.toList()));
Review comment:
Timeline Instants= -> Timeline instants =
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] lw309637554 commented on a change in pull request #3259: [HUDI-2164] Let users build cluster plan and execute this plan at once using HoodieClusteringJob for async clustering
lw309637554 commented on a change in pull request #3259:
URL: https://github.com/apache/hudi/pull/3259#discussion_r675270601
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieClusteringJob.java
##
@@ -164,11 +196,37 @@ private int doCluster(JavaSparkContext jsc) throws
Exception {
private Option doSchedule(JavaSparkContext jsc) throws Exception {
String schemaStr = getSchemaFromLatestInstant();
try (SparkRDDWriteClient client =
UtilHelpers.createHoodieClient(jsc, cfg.basePath, schemaStr, cfg.parallelism,
Option.empty(), props)) {
- if (cfg.clusteringInstantTime != null) {
-client.scheduleClusteringAtInstant(cfg.clusteringInstantTime,
Option.empty());
-return Option.of(cfg.clusteringInstantTime);
+ return doSchedule(client);
+}
+ }
+
+ private Option doSchedule(SparkRDDWriteClient
client) {
+if (cfg.clusteringInstantTime != null) {
+ client.scheduleClusteringAtInstant(cfg.clusteringInstantTime,
Option.empty());
+ return Option.of(cfg.clusteringInstantTime);
+}
+return client.scheduleClustering(Option.empty());
+ }
+
+ public int doScheduleAndCluster(JavaSparkContext jsc) throws Exception {
+LOG.info("Step 1: Do schedule");
+String schemaStr = getSchemaFromLatestInstant();
+try (SparkRDDWriteClient client =
UtilHelpers.createHoodieClient(jsc, cfg.basePath, schemaStr, cfg.parallelism,
Option.empty(), props)) {
+
+ Option instantTime = doSchedule(client);
+ int result = instantTime.isPresent() ? 0 : -1;
+
+ if (result == -1) {
+LOG.info("Couldn't Generate Cluster Plan");
Review comment:
Couldn't Generate Cluster Plan -> Couldn't generate cluster plan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[hudi] branch master updated (c89bf1d -> 5a2f3d4)
This is an automated email from the ASF dual-hosted git repository. zhiwei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from c89bf1d [HUDI-2205] Rollback inflight compaction for flink writer (#3320) add 5a2f3d4 [HUDI-2139] MergeInto MOR Table May Result InCorrect Result (#3230) No new revisions were added by this update. Summary of changes: .../org/apache/hudi/io/HoodieAppendHandle.java | 12 +- .../hudi/common/model/HoodiePayloadProps.java | 10 ++ .../hudi/command/MergeIntoHoodieTableCommand.scala | 23 ++-- .../hudi/command/payload/ExpressionPayload.scala | 108 ++-- .../spark/sql/hudi/TestMereIntoLogOnlyTable.scala | 2 +- .../spark/sql/hudi/TestMergeIntoTable2.scala | 138 + 6 files changed, 241 insertions(+), 52 deletions(-) create mode 100644 hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable2.scala
[jira] [Commented] (HUDI-2101) support z-order for hudi
[ https://issues.apache.org/jira/browse/HUDI-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385867#comment-17385867 ] ASF GitHub Bot commented on HUDI-2101: -- hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 5a7e153b3b8cdf2d5922839db356282b01dc8d92 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1109) * 93897c10f0635eb163c9c98894092b034e3fcb14 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > support z-order for hudi > > > Key: HUDI-2101 > URL: https://issues.apache.org/jira/browse/HUDI-2101 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: tao meng >Assignee: tao meng >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > support z-order for hudi to optimze the query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2139) MergeInto MOR Table May Result InCorrect Result
[
https://issues.apache.org/jira/browse/HUDI-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385868#comment-17385868
]
ASF GitHub Bot commented on HUDI-2139:
--
pengzhiwei2018 merged pull request #3230:
URL: https://github.com/apache/hudi/pull/3230
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MergeInto MOR Table May Result InCorrect Result
> ---
>
> Key: HUDI-2139
> URL: https://issues.apache.org/jira/browse/HUDI-2139
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Currently we process all the update-action and inert-action in the
> ExpressionPayload#
> getInsertValue without know whether the record is matched or not matched for
> MOR table. This may result in incorrect merge result. e.g.
> {code:java}
> Merge into h0
> using (select 2 as id, 'a1' as name, 10 as price from s) s0
> on h0.id = s0.id
> when matched then s0.id = 1 the update set id = s0.id, name = s0.name, price
> = 10
> when not matched then s0.id = 2 the insert (id,name,price) values(id,name,
> 20){code}
> If the id = 2 can matched the target table h0, but it cannot match the
> udpate-condition ( s0.id = 1), It should not update the table. However,
> currently we cannot know the matched state of the input record, it will goes
> to the not-matched actions and update the price to 20 finally. This is
> incorrect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] pengzhiwei2018 merged pull request #3230: [HUDI-2139] MergeInto MOR Table May Result InCorrect Result
pengzhiwei2018 merged pull request #3230: URL: https://github.com/apache/hudi/pull/3230 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3330: [HUDI-2101][WIP]support z-order for hudi
hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 5a7e153b3b8cdf2d5922839db356282b01dc8d92 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1109) * 93897c10f0635eb163c9c98894092b034e3fcb14 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2101) support z-order for hudi
[ https://issues.apache.org/jira/browse/HUDI-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385866#comment-17385866 ] ASF GitHub Bot commented on HUDI-2101: -- hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 5a7e153b3b8cdf2d5922839db356282b01dc8d92 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1109) * 93897c10f0635eb163c9c98894092b034e3fcb14 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > support z-order for hudi > > > Key: HUDI-2101 > URL: https://issues.apache.org/jira/browse/HUDI-2101 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: tao meng >Assignee: tao meng >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > support z-order for hudi to optimze the query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot edited a comment on pull request #3330: [HUDI-2101][WIP]support z-order for hudi
hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 5a7e153b3b8cdf2d5922839db356282b01dc8d92 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1109) * 93897c10f0635eb163c9c98894092b034e3fcb14 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2139) MergeInto MOR Table May Result InCorrect Result
[
https://issues.apache.org/jira/browse/HUDI-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385865#comment-17385865
]
ASF GitHub Bot commented on HUDI-2139:
--
pengzhiwei2018 commented on a change in pull request #3230:
URL: https://github.com/apache/hudi/pull/3230#discussion_r675268629
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java
##
@@ -108,13 +110,16 @@
protected final Map header = new HashMap<>();
private SizeEstimator sizeEstimator;
+ private Properties recordProperties = new Properties();
+
public HoodieAppendHandle(HoodieWriteConfig config, String instantTime,
HoodieTable hoodieTable,
String partitionPath, String fileId,
Iterator> recordItr, TaskContextSupplier taskContextSupplier) {
super(config, instantTime, partitionPath, fileId, hoodieTable,
taskContextSupplier);
this.fileId = fileId;
this.recordItr = recordItr;
sizeEstimator = new DefaultSizeEstimator();
this.statuses = new ArrayList<>();
+this.recordProperties.putAll(config.getProps());
Review comment:
Because I do not want to affect the origin config which may used in some
other place. So I make a copy for the HoodieAppendHandle. It is more safe after
the copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MergeInto MOR Table May Result InCorrect Result
> ---
>
> Key: HUDI-2139
> URL: https://issues.apache.org/jira/browse/HUDI-2139
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Currently we process all the update-action and inert-action in the
> ExpressionPayload#
> getInsertValue without know whether the record is matched or not matched for
> MOR table. This may result in incorrect merge result. e.g.
> {code:java}
> Merge into h0
> using (select 2 as id, 'a1' as name, 10 as price from s) s0
> on h0.id = s0.id
> when matched then s0.id = 1 the update set id = s0.id, name = s0.name, price
> = 10
> when not matched then s0.id = 2 the insert (id,name,price) values(id,name,
> 20){code}
> If the id = 2 can matched the target table h0, but it cannot match the
> udpate-condition ( s0.id = 1), It should not update the table. However,
> currently we cannot know the matched state of the input record, it will goes
> to the not-matched actions and update the price to 20 finally. This is
> incorrect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3230: [HUDI-2139] MergeInto MOR Table May Result InCorrect Result
pengzhiwei2018 commented on a change in pull request #3230:
URL: https://github.com/apache/hudi/pull/3230#discussion_r675268629
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java
##
@@ -108,13 +110,16 @@
protected final Map header = new HashMap<>();
private SizeEstimator sizeEstimator;
+ private Properties recordProperties = new Properties();
+
public HoodieAppendHandle(HoodieWriteConfig config, String instantTime,
HoodieTable hoodieTable,
String partitionPath, String fileId,
Iterator> recordItr, TaskContextSupplier taskContextSupplier) {
super(config, instantTime, partitionPath, fileId, hoodieTable,
taskContextSupplier);
this.fileId = fileId;
this.recordItr = recordItr;
sizeEstimator = new DefaultSizeEstimator();
this.statuses = new ArrayList<>();
+this.recordProperties.putAll(config.getProps());
Review comment:
Because I do not want to affect the origin config which may used in some
other place. So I make a copy for the HoodieAppendHandle. It is more safe after
the copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-2139) MergeInto MOR Table May Result InCorrect Result
[
https://issues.apache.org/jira/browse/HUDI-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385864#comment-17385864
]
ASF GitHub Bot commented on HUDI-2139:
--
pengzhiwei2018 commented on a change in pull request #3230:
URL: https://github.com/apache/hudi/pull/3230#discussion_r675268629
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java
##
@@ -108,13 +110,16 @@
protected final Map header = new HashMap<>();
private SizeEstimator sizeEstimator;
+ private Properties recordProperties = new Properties();
+
public HoodieAppendHandle(HoodieWriteConfig config, String instantTime,
HoodieTable hoodieTable,
String partitionPath, String fileId,
Iterator> recordItr, TaskContextSupplier taskContextSupplier) {
super(config, instantTime, partitionPath, fileId, hoodieTable,
taskContextSupplier);
this.fileId = fileId;
this.recordItr = recordItr;
sizeEstimator = new DefaultSizeEstimator();
this.statuses = new ArrayList<>();
+this.recordProperties.putAll(config.getProps());
Review comment:
Because I do not want to affect the origin config which may used in some
other place. So I make a copy for the HoodieAppendHandle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MergeInto MOR Table May Result InCorrect Result
> ---
>
> Key: HUDI-2139
> URL: https://issues.apache.org/jira/browse/HUDI-2139
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
>Reporter: pengzhiwei
>Assignee: pengzhiwei
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Currently we process all the update-action and inert-action in the
> ExpressionPayload#
> getInsertValue without know whether the record is matched or not matched for
> MOR table. This may result in incorrect merge result. e.g.
> {code:java}
> Merge into h0
> using (select 2 as id, 'a1' as name, 10 as price from s) s0
> on h0.id = s0.id
> when matched then s0.id = 1 the update set id = s0.id, name = s0.name, price
> = 10
> when not matched then s0.id = 2 the insert (id,name,price) values(id,name,
> 20){code}
> If the id = 2 can matched the target table h0, but it cannot match the
> udpate-condition ( s0.id = 1), It should not update the table. However,
> currently we cannot know the matched state of the input record, it will goes
> to the not-matched actions and update the price to 20 finally. This is
> incorrect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-2164) Build cluster plan and execute this plan at once for HoodieClusteringJob
[
https://issues.apache.org/jira/browse/HUDI-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385863#comment-17385863
]
ASF GitHub Bot commented on HUDI-2164:
--
lw309637554 commented on a change in pull request #3259:
URL: https://github.com/apache/hudi/pull/3259#discussion_r675268621
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieClusteringJob.java
##
@@ -121,17 +141,29 @@ public static void main(String[] args) {
public int cluster(int retry) {
this.fs = FSUtils.getFs(cfg.basePath, jsc.hadoopConfiguration());
int ret = UtilHelpers.retry(retry, () -> {
- if (cfg.runSchedule) {
-LOG.info("Do schedule");
-Option instantTime = doSchedule(jsc);
-int result = instantTime.isPresent() ? 0 : -1;
-if (result == 0) {
- LOG.info("The schedule instant time is " + instantTime.get());
+ String runningMode = cfg.runningMode == null ? "" :
cfg.runningMode.toLowerCase();
Review comment:
"" -> can we use Optional to check null? "" is confused
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Build cluster plan and execute this plan at once for HoodieClusteringJob
>
>
> Key: HUDI-2164
> URL: https://issues.apache.org/jira/browse/HUDI-2164
> Project: Apache Hudi
> Issue Type: Task
>Reporter: Yue Zhang
>Priority: Major
> Labels: pull-request-available
>
> For now, Hudi can let users submit a HoodieClusteringJob to build a
> clustering plan or execute a clustering plan through --schedule or
> --instant-time config.
> If users want to trigger a clustering job, he has to
> # Submit a HoodieClusteringJob to build a clustering job through --schedule
> config
> # Copy the created clustering Instant time form Log info.
> # Submit the HoodieClusteringJob again to execute this created clustering
> plan through --instant-time config.
> The pain point is that there are too many steps when trigger a clustering and
> need to copy and paste the instant time from log file manually so that we
> can't make it automatically.
>
> I just raise a PR to offer a new config named --mode or -m in short
> ||--mode||remarks||
> |execute|Execute a cluster plan at given instant which means --instant-time
> is needed here. default value. |
> |schedule|Make a clustering plan.|
> |*scheduleAndExecute*|Make a cluster plan first and execute that plan
> immediately|
> Now users can use --mode scheduleAndExecute to Build cluster plan and execute
> this plan at once using HoodieClusteringJob.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3230: [HUDI-2139] MergeInto MOR Table May Result InCorrect Result
pengzhiwei2018 commented on a change in pull request #3230:
URL: https://github.com/apache/hudi/pull/3230#discussion_r675268629
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java
##
@@ -108,13 +110,16 @@
protected final Map header = new HashMap<>();
private SizeEstimator sizeEstimator;
+ private Properties recordProperties = new Properties();
+
public HoodieAppendHandle(HoodieWriteConfig config, String instantTime,
HoodieTable hoodieTable,
String partitionPath, String fileId,
Iterator> recordItr, TaskContextSupplier taskContextSupplier) {
super(config, instantTime, partitionPath, fileId, hoodieTable,
taskContextSupplier);
this.fileId = fileId;
this.recordItr = recordItr;
sizeEstimator = new DefaultSizeEstimator();
this.statuses = new ArrayList<>();
+this.recordProperties.putAll(config.getProps());
Review comment:
Because I do not want to affect the origin config which may used in some
other place. So I make a copy for the HoodieAppendHandle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] lw309637554 commented on a change in pull request #3259: [HUDI-2164] Let users build cluster plan and execute this plan at once using HoodieClusteringJob for async clustering
lw309637554 commented on a change in pull request #3259:
URL: https://github.com/apache/hudi/pull/3259#discussion_r675268621
##
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieClusteringJob.java
##
@@ -121,17 +141,29 @@ public static void main(String[] args) {
public int cluster(int retry) {
this.fs = FSUtils.getFs(cfg.basePath, jsc.hadoopConfiguration());
int ret = UtilHelpers.retry(retry, () -> {
- if (cfg.runSchedule) {
-LOG.info("Do schedule");
-Option instantTime = doSchedule(jsc);
-int result = instantTime.isPresent() ? 0 : -1;
-if (result == 0) {
- LOG.info("The schedule instant time is " + instantTime.get());
+ String runningMode = cfg.runningMode == null ? "" :
cfg.runningMode.toLowerCase();
Review comment:
"" -> can we use Optional to check null? "" is confused
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3331: [HUDI-2211] Fix NullPointerException in TestHoodieConsoleMetrics
hudi-bot edited a comment on pull request #3331: URL: https://github.com/apache/hudi/pull/3331#issuecomment-885351519 ## CI report: * fa7d1d55847d21fb5d89451b1785c479ec77554d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1110) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2211) Fix NullPointerException in TestHoodieConsoleMetrics
[ https://issues.apache.org/jira/browse/HUDI-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385861#comment-17385861 ] ASF GitHub Bot commented on HUDI-2211: -- hudi-bot edited a comment on pull request #3331: URL: https://github.com/apache/hudi/pull/3331#issuecomment-885351519 ## CI report: * fa7d1d55847d21fb5d89451b1785c479ec77554d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1110) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix NullPointerException in TestHoodieConsoleMetrics > > > Key: HUDI-2211 > URL: https://issues.apache.org/jira/browse/HUDI-2211 > Project: Apache Hudi > Issue Type: Improvement > Components: metrics >Reporter: Xuedong Luan >Assignee: Xuedong Luan >Priority: Minor > Labels: pull-request-available > > java.lang.NullPointerException: Expected a non-null value. Got > nulljava.lang.NullPointerException: Expected a non-null value. Got null at > org.apache.hudi.common.util.Option.(Option.java:65) at > org.apache.hudi.common.util.Option.of(Option.java:75) at > org.apache.hudi.metrics.Metrics.registerHoodieCommonMetrics(Metrics.java:85) > at org.apache.hudi.metrics.Metrics.reportAndCloseReporter(Metrics.java:63) at > org.apache.hudi.metrics.Metrics.shutdown(Metrics.java:109) at > org.apache.hudi.metrics.TestHoodieConsoleMetrics.stop(TestHoodieConsoleMetrics.java:48) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2164) Build cluster plan and execute this plan at once for HoodieClusteringJob
[ https://issues.apache.org/jira/browse/HUDI-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385860#comment-17385860 ] ASF GitHub Bot commented on HUDI-2164: -- lw309637554 commented on a change in pull request #3259: URL: https://github.com/apache/hudi/pull/3259#discussion_r675267730 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieClusteringJob.java ## @@ -49,6 +51,9 @@ private transient FileSystem fs; private TypedProperties props; private final JavaSparkContext jsc; + private static final String EXECUTE = "execute"; + private static final String SCHEDULE = "schedule"; + private static final String SCHEDULE_AND_EXECUTE = "scheduleandexecute"; Review comment: is scheduleandexecute -> scheduleAndExecute better? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Build cluster plan and execute this plan at once for HoodieClusteringJob > > > Key: HUDI-2164 > URL: https://issues.apache.org/jira/browse/HUDI-2164 > Project: Apache Hudi > Issue Type: Task >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > > For now, Hudi can let users submit a HoodieClusteringJob to build a > clustering plan or execute a clustering plan through --schedule or > --instant-time config. > If users want to trigger a clustering job, he has to > # Submit a HoodieClusteringJob to build a clustering job through --schedule > config > # Copy the created clustering Instant time form Log info. > # Submit the HoodieClusteringJob again to execute this created clustering > plan through --instant-time config. > The pain point is that there are too many steps when trigger a clustering and > need to copy and paste the instant time from log file manually so that we > can't make it automatically. > > I just raise a PR to offer a new config named --mode or -m in short > ||--mode||remarks|| > |execute|Execute a cluster plan at given instant which means --instant-time > is needed here. default value. | > |schedule|Make a clustering plan.| > |*scheduleAndExecute*|Make a cluster plan first and execute that plan > immediately| > Now users can use --mode scheduleAndExecute to Build cluster plan and execute > this plan at once using HoodieClusteringJob. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] lw309637554 commented on a change in pull request #3259: [HUDI-2164] Let users build cluster plan and execute this plan at once using HoodieClusteringJob for async clustering
lw309637554 commented on a change in pull request #3259: URL: https://github.com/apache/hudi/pull/3259#discussion_r675267730 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieClusteringJob.java ## @@ -49,6 +51,9 @@ private transient FileSystem fs; private TypedProperties props; private final JavaSparkContext jsc; + private static final String EXECUTE = "execute"; + private static final String SCHEDULE = "schedule"; + private static final String SCHEDULE_AND_EXECUTE = "scheduleandexecute"; Review comment: is scheduleandexecute -> scheduleAndExecute better? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[
https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385859#comment-17385859
]
ASF GitHub Bot commented on HUDI-2176:
--
nsivabalan commented on a change in pull request #3306:
URL: https://github.com/apache/hudi/pull/3306#discussion_r675267422
##
File path:
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java
##
@@ -94,6 +101,18 @@ public BaseSparkCommitActionExecutor(HoodieEngineContext
context,
WriteOperationType operationType,
Option extraMetadata) {
super(context, config, table, instantTime, operationType, extraMetadata);
+initKeyGenIfNeeded();
+ }
+
+ private void initKeyGenIfNeeded() {
+this.populateMetaFields = config.populateMetaFields();
+if (!populateMetaFields) {
+ try {
+keyGeneratorOpt = Option.of((BaseKeyGenerator)
HoodieSparkKeyGeneratorFactory.createKeyGenerator(new
TypedProperties(config.getProps(;
+ } catch (IOException e) {
+throw new HoodieIOException("Only BaseKeyGenerators are supported when
meta columns are disabled ", e);
Review comment:
But I added as a guard for catching casting issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Virutal keys support for COW all operations
> ---
>
> Key: HUDI-2176
> URL: https://issues.apache.org/jira/browse/HUDI-2176
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Writer Core
>Reporter: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Virutal keys support for COW all operations
> (merge handle)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hudi] nsivabalan commented on a change in pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table
nsivabalan commented on a change in pull request #3306:
URL: https://github.com/apache/hudi/pull/3306#discussion_r675267422
##
File path:
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java
##
@@ -94,6 +101,18 @@ public BaseSparkCommitActionExecutor(HoodieEngineContext
context,
WriteOperationType operationType,
Option extraMetadata) {
super(context, config, table, instantTime, operationType, extraMetadata);
+initKeyGenIfNeeded();
+ }
+
+ private void initKeyGenIfNeeded() {
+this.populateMetaFields = config.populateMetaFields();
+if (!populateMetaFields) {
+ try {
+keyGeneratorOpt = Option.of((BaseKeyGenerator)
HoodieSparkKeyGeneratorFactory.createKeyGenerator(new
TypedProperties(config.getProps(;
+ } catch (IOException e) {
+throw new HoodieIOException("Only BaseKeyGenerators are supported when
meta columns are disabled ", e);
Review comment:
But I added as a guard for catching casting issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-2101) support z-order for hudi
[ https://issues.apache.org/jira/browse/HUDI-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385855#comment-17385855 ] ASF GitHub Bot commented on HUDI-2101: -- hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 5a7e153b3b8cdf2d5922839db356282b01dc8d92 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1109) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > support z-order for hudi > > > Key: HUDI-2101 > URL: https://issues.apache.org/jira/browse/HUDI-2101 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: tao meng >Assignee: tao meng >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > support z-order for hudi to optimze the query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-2211) Fix NullPointerException in TestHoodieConsoleMetrics
[ https://issues.apache.org/jira/browse/HUDI-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385856#comment-17385856 ] ASF GitHub Bot commented on HUDI-2211: -- hudi-bot commented on pull request #3331: URL: https://github.com/apache/hudi/pull/3331#issuecomment-885351519 ## CI report: * fa7d1d55847d21fb5d89451b1785c479ec77554d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix NullPointerException in TestHoodieConsoleMetrics > > > Key: HUDI-2211 > URL: https://issues.apache.org/jira/browse/HUDI-2211 > Project: Apache Hudi > Issue Type: Improvement > Components: metrics >Reporter: Xuedong Luan >Assignee: Xuedong Luan >Priority: Minor > Labels: pull-request-available > > java.lang.NullPointerException: Expected a non-null value. Got > nulljava.lang.NullPointerException: Expected a non-null value. Got null at > org.apache.hudi.common.util.Option.(Option.java:65) at > org.apache.hudi.common.util.Option.of(Option.java:75) at > org.apache.hudi.metrics.Metrics.registerHoodieCommonMetrics(Metrics.java:85) > at org.apache.hudi.metrics.Metrics.reportAndCloseReporter(Metrics.java:63) at > org.apache.hudi.metrics.Metrics.shutdown(Metrics.java:109) at > org.apache.hudi.metrics.TestHoodieConsoleMetrics.stop(TestHoodieConsoleMetrics.java:48) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] hudi-bot commented on pull request #3331: [HUDI-2211] Fix NullPointerException in TestHoodieConsoleMetrics
hudi-bot commented on pull request #3331: URL: https://github.com/apache/hudi/pull/3331#issuecomment-885351519 ## CI report: * fa7d1d55847d21fb5d89451b1785c479ec77554d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3330: [HUDI-2101][WIP]support z-order for hudi
hudi-bot edited a comment on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885350571 ## CI report: * 5a7e153b3b8cdf2d5922839db356282b01dc8d92 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1109) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2101) support z-order for hudi
[ https://issues.apache.org/jira/browse/HUDI-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385852#comment-17385852 ] ASF GitHub Bot commented on HUDI-2101: -- xiarixiaoyao commented on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885351003 now the RFC-27 is not implement, once RFC-27 is merged, i will update code to adapt it. in this pr, even if we donnot do data skipping, we can also achive a good result, since we sort data by z-order hilbert implement will come soon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > support z-order for hudi > > > Key: HUDI-2101 > URL: https://issues.apache.org/jira/browse/HUDI-2101 > Project: Apache Hudi > Issue Type: Sub-task > Components: Spark Integration >Reporter: tao meng >Assignee: tao meng >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > support z-order for hudi to optimze the query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385853#comment-17385853 ] ASF GitHub Bot commented on HUDI-648: - lw309637554 commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-885351154 @liujinhui1994 hello, does it ready to review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hudi] lw309637554 commented on pull request #3312: [HUDI-648][RFC-20] Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
lw309637554 commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-885351154 @liujinhui1994 hello, does it ready to review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3330: [HUDI-2101][WIP]support z-order for hudi
xiarixiaoyao commented on pull request #3330: URL: https://github.com/apache/hudi/pull/3330#issuecomment-885351003 now the RFC-27 is not implement, once RFC-27 is merged, i will update code to adapt it. in this pr, even if we donnot do data skipping, we can also achive a good result, since we sort data by z-order hilbert implement will come soon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations
[
https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385850#comment-17385850
]
ASF GitHub Bot commented on HUDI-2176:
--
vinothchandar commented on a change in pull request #3306:
URL: https://github.com/apache/hudi/pull/3306#discussion_r675266638
##
File path:
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java
##
@@ -94,6 +101,18 @@ public BaseSparkCommitActionExecutor(HoodieEngineContext
context,
WriteOperationType operationType,
Option extraMetadata) {
super(context, config, table, instantTime, operationType, extraMetadata);
+initKeyGenIfNeeded();
+ }
+
+ private void initKeyGenIfNeeded() {
+this.populateMetaFields = config.populateMetaFields();
+if (!populateMetaFields) {
+ try {
+keyGeneratorOpt = Option.of((BaseKeyGenerator)
HoodieSparkKeyGeneratorFactory.createKeyGenerator(new
TypedProperties(config.getProps(;
+ } catch (IOException e) {
+throw new HoodieIOException("Only BaseKeyGenerators are supported when
meta columns are disabled ", e);
Review comment:
>within HoodieSparkKeyGeneratorFactory.createKeyGenerator.
yes. within. You can still cast outside right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Virutal keys support for COW all operations
> ---
>
> Key: HUDI-2176
> URL: https://issues.apache.org/jira/browse/HUDI-2176
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Writer Core
>Reporter: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Virutal keys support for COW all operations
> (merge handle)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
