[jira] [Updated] (HUDI-2837) The original hoodie.table.name should be maintained in Spark SQL

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2837: - Sprint: (was: Hudi-Sprint-0.10.1) > The original hoodie.table.name should be maintained in Spark SQL >

[jira] [Updated] (HUDI-2966) Add TaskCompletionListener for HoodieMergeOnReadRDD to close logScanner

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2966: - Sprint: Hudi-Sprint-0.10.1 > Add TaskCompletionListener for HoodieMergeOnReadRDD to close logScanner >

[jira] [Updated] (HUDI-2837) The original hoodie.table.name should be maintained in Spark SQL

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2837: - Fix Version/s: (was: 0.10.1) > The original hoodie.table.name should be maintained in Spark SQL >

[jira] [Updated] (HUDI-281) HiveSync failure through Spark when useJdbc is set to false

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-281: Fix Version/s: 0.10.1 > HiveSync failure through Spark when useJdbc is set to false >

[jira] [Updated] (HUDI-281) HiveSync failure through Spark when useJdbc is set to false

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-281: Sprint: Hudi-Sprint-0.10.1 > HiveSync failure through Spark when useJdbc is set to false >

[GitHub] [hudi] LuPan2015 commented on issue #4474: [SUPPORT] Should we shade all aws dependencies to avoid class conflicts?

2021-12-29 Thread GitBox
LuPan2015 commented on issue #4474: URL: https://github.com/apache/hudi/issues/4474#issuecomment-1002855242 I seem to have encountered the same problem #4475 . Is there any good way to solve it quickly? -- This is an automated message from the Apache Git Service. To respond to the

[jira] [Resolved] (HUDI-3083) Support component data types for flink bulk_insert

2021-12-29 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-3083. -- > Support component data types for flink bulk_insert > -- >

[GitHub] [hudi] LuPan2015 opened a new issue #4475: [SUPPORT] Hudi and aws S3 integration exception

2021-12-29 Thread GitBox
LuPan2015 opened a new issue #4475: URL: https://github.com/apache/hudi/issues/4475 **Describe the problem you faced** The Hudi table is created not successfully and the data is stored in S3 **To Reproduce** Steps to reproduce the behavior: 1. Configure

[jira] [Commented] (HUDI-3083) Support component data types for flink bulk_insert

2021-12-29 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1746#comment-1746 ] Danny Chen commented on HUDI-3083: -- Fixed via master branch: 674c1492348b5b2a93358c9dd51a1adfe6a8ecf2 >

[GitHub] [hudi] danny0405 merged pull request #4470: [HUDI-3083] Support component data types for flink bulk_insert

2021-12-29 Thread GitBox
danny0405 merged pull request #4470: URL: https://github.com/apache/hudi/pull/4470 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[hudi] branch master updated: [HUDI-3083] Support component data types for flink bulk_insert (#4470)

2021-12-29 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 674c149 [HUDI-3083] Support component data

[GitHub] [hudi] hudi-bot removed a comment on pull request #4470: [HUDI-3083] Support component data types for flink bulk_insert

2021-12-29 Thread GitBox
hudi-bot removed a comment on pull request #4470: URL: https://github.com/apache/hudi/pull/4470#issuecomment-1002837480 ## CI report: * f718cea8b001eaeb132d97b2ae0e49f1bc101c0f Azure:

[GitHub] [hudi] hudi-bot commented on pull request #4470: [HUDI-3083] Support component data types for flink bulk_insert

2021-12-29 Thread GitBox
hudi-bot commented on pull request #4470: URL: https://github.com/apache/hudi/pull/4470#issuecomment-1002850453 ## CI report: * 4323140d1fbf1e2f0177671268ca7f5e7473a4bb Azure:

[GitHub] [hudi] hudi-bot commented on pull request #4473: [HUDI-2590] Adding tests to validate different key generators

2021-12-29 Thread GitBox
hudi-bot commented on pull request #4473: URL: https://github.com/apache/hudi/pull/4473#issuecomment-1002849419 ## CI report: * 35841cdbffb0edd8d7e1f114147b12ee3daf0872 UNKNOWN * b91d9c4a42a05e01ee5a75449e861d9bf88b69c7 Azure:

[GitHub] [hudi] hudi-bot removed a comment on pull request #4473: [HUDI-2590] Adding tests to validate different key generators

2021-12-29 Thread GitBox
hudi-bot removed a comment on pull request #4473: URL: https://github.com/apache/hudi/pull/4473#issuecomment-1002848920 ## CI report: * 35841cdbffb0edd8d7e1f114147b12ee3daf0872 UNKNOWN * b91d9c4a42a05e01ee5a75449e861d9bf88b69c7 UNKNOWN Bot commands @hudi-bot

[GitHub] [hudi] hudi-bot commented on pull request #4473: [HUDI-2590] Adding tests to validate different key generators

2021-12-29 Thread GitBox
hudi-bot commented on pull request #4473: URL: https://github.com/apache/hudi/pull/4473#issuecomment-1002848920 ## CI report: * 35841cdbffb0edd8d7e1f114147b12ee3daf0872 UNKNOWN * b91d9c4a42a05e01ee5a75449e861d9bf88b69c7 UNKNOWN Bot commands @hudi-bot supports

[GitHub] [hudi] hudi-bot removed a comment on pull request #4473: [HUDI-2590] Adding tests to validate different key generators

2021-12-29 Thread GitBox
hudi-bot removed a comment on pull request #4473: URL: https://github.com/apache/hudi/pull/4473#issuecomment-1002848366 ## CI report: * 35841cdbffb0edd8d7e1f114147b12ee3daf0872 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot

[GitHub] [hudi] danny0405 merged pull request #4465: [HUDI-2959] Reverting previous revert of HUDI-2959 Original PR fixed a leak in async service. Reverted due to CI flakiness.

2021-12-29 Thread GitBox
danny0405 merged pull request #4465: URL: https://github.com/apache/hudi/pull/4465 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[hudi] branch master updated (504747e -> 5c0e4ce)

2021-12-29 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 504747e [HUDI-3108] Fix Purge Drop MOR Table Cause error (#4455) add 5c0e4ce Revert "[HUDI-3043] Revert

[GitHub] [hudi] boneanxs opened a new issue #4474: [SUPPORT] Should we shade all aws dependencies to avoid class conflicts?

2021-12-29 Thread GitBox
boneanxs opened a new issue #4474: URL: https://github.com/apache/hudi/issues/4474 As we introduce support for DynamoDb based lock by [HUDI-2314](https://github.com/apache/hudi/pull/3486/files), can we shade all aws dependencies for all our bundled jars(spark, flink)? As many users also

[GitHub] [hudi] hudi-bot commented on pull request #4473: [HUDI-2590] Adding tests to validate different key generators

2021-12-29 Thread GitBox
hudi-bot commented on pull request #4473: URL: https://github.com/apache/hudi/pull/4473#issuecomment-1002848366 ## CI report: * 35841cdbffb0edd8d7e1f114147b12ee3daf0872 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure`

[GitHub] [hudi] nsivabalan opened a new pull request #4473: [HUDI-2590] Adding tests to validate different key generators

2021-12-29 Thread GitBox
nsivabalan opened a new pull request #4473: URL: https://github.com/apache/hudi/pull/4473 ## What is the purpose of the pull request Added tests to validate COW table for different queries for different key generators ## Brief change log *(for example:)* - *Modify

[GitHub] [hudi] hudi-bot commented on pull request #3877: [HUDI-2590][WIP] Test different keygen with and without glob path

2021-12-29 Thread GitBox
hudi-bot commented on pull request #3877: URL: https://github.com/apache/hudi/pull/3877#issuecomment-1002846885 ## CI report: * 2e115bcaa51b20357eb110e5f30e18098c474997 Azure:

[GitHub] [hudi] hudi-bot removed a comment on pull request #3877: [HUDI-2590][WIP] Test different keygen with and without glob path

2021-12-29 Thread GitBox
hudi-bot removed a comment on pull request #3877: URL: https://github.com/apache/hudi/pull/3877#issuecomment-1002837802 ## CI report: * c721ba387780ecac09feb45b79d23c98b9d4d39a Azure:

[jira] [Resolved] (HUDI-3108) Fix Purge Drop MOR Table Cause error

2021-12-29 Thread Forward Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Forward Xu resolved HUDI-3108. -- > Fix Purge Drop MOR Table Cause error > - > > Key:

[GitHub] [hudi] XuQianJin-Stars commented on a change in pull request #4471: [HUDI-3125] spark-sql write timestamp directly

2021-12-29 Thread GitBox
XuQianJin-Stars commented on a change in pull request #4471: URL: https://github.com/apache/hudi/pull/4471#discussion_r776548636 ## File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionHelper.scala ## @@ -301,9 +302,17 @@ object

[jira] [Updated] (HUDI-2465) Fix merge, update for spark sql dml support to test suite infra

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2465: - Fix Version/s: 0.10.1 > Fix merge, update for spark sql dml support to test suite infra >

[jira] [Updated] (HUDI-2661) java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.catalog.CatalogTable.copy

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2661: - Sprint: Hudi-Sprint-0.10.1 > java.lang.NoSuchMethodError: >

[jira] [Updated] (HUDI-3100) Hive Conditional sync cannot be set from deltastreamer

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3100: - Sprint: Hudi-Sprint-0.10.1 > Hive Conditional sync cannot be set from deltastreamer >

[jira] [Updated] (HUDI-2611) `create table if not exists` should print message instead of throwing error

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2611: - Sprint: Hudi-Sprint-0.10.1 > `create table if not exists` should print message instead of throwing error

[jira] [Updated] (HUDI-1850) Read on table fails if the first write to table failed

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-1850: - Sprint: Hudi-Sprint-0.10.1 > Read on table fails if the first write to table failed >

[jira] [Updated] (HUDI-2915) Fix field not found in record error for spark-sql

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2915: - Sprint: Hudi-Sprint-0.10.1 > Fix field not found in record error for spark-sql >

[jira] [Updated] (HUDI-2837) The original hoodie.table.name should be maintained in Spark SQL

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2837: - Sprint: Hudi-Sprint-0.10.1 > The original hoodie.table.name should be maintained in Spark SQL >

[jira] [Updated] (HUDI-2426) spark sql extensions breaks read.table from metastore

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2426: - Sprint: Hudi-Sprint-0.10.1 > spark sql extensions breaks read.table from metastore >

[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #4468: [Issue: #2802] Fixing Hive getSchema for RT tables

2021-12-29 Thread GitBox
xiarixiaoyao commented on a change in pull request #4468: URL: https://github.com/apache/hudi/pull/4468#discussion_r776545682 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/AbstractRealtimeRecordReader.java ## @@ -77,19 +74,17 @@ private boolean

[jira] [Updated] (HUDI-3125) Spark SQL writing timestamp type don't need to disable `spark.sql.datetime.java8API.enabled` manually

2021-12-29 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3125: - Sprint: Hudi-Sprint-0.10.1 > Spark SQL writing timestamp type don't need to disable >

[hudi] branch asf-site updated: [MINOR] - Publishing blog for Zorder (#4472)

2021-12-29 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 269d3bc [MINOR] - Publishing blog for Zorder

[GitHub] [hudi] vinothchandar merged pull request #4472: [MINOR] - Publishing blog for Zorder

2021-12-29 Thread GitBox
vinothchandar merged pull request #4472: URL: https://github.com/apache/hudi/pull/4472 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] hudi-bot commented on pull request #3877: [HUDI-2590][WIP] Test different keygen with and without glob path

2021-12-29 Thread GitBox
hudi-bot commented on pull request #3877: URL: https://github.com/apache/hudi/pull/3877#issuecomment-1002837802 ## CI report: * c721ba387780ecac09feb45b79d23c98b9d4d39a Azure:

[GitHub] [hudi] hudi-bot removed a comment on pull request #3877: [HUDI-2590][WIP] Test different keygen with and without glob path

2021-12-29 Thread GitBox
hudi-bot removed a comment on pull request #3877: URL: https://github.com/apache/hudi/pull/3877#issuecomment-1002837366 ## CI report: * c721ba387780ecac09feb45b79d23c98b9d4d39a Azure:

[GitHub] [hudi] hudi-bot removed a comment on pull request #4470: [HUDI-3083] Support component data types for flink bulk_insert

2021-12-29 Thread GitBox
hudi-bot removed a comment on pull request #4470: URL: https://github.com/apache/hudi/pull/4470#issuecomment-1002837046 ## CI report: * f718cea8b001eaeb132d97b2ae0e49f1bc101c0f Azure:

[GitHub] [hudi] hudi-bot commented on pull request #4470: [HUDI-3083] Support component data types for flink bulk_insert

2021-12-29 Thread GitBox
hudi-bot commented on pull request #4470: URL: https://github.com/apache/hudi/pull/4470#issuecomment-1002837480 ## CI report: * f718cea8b001eaeb132d97b2ae0e49f1bc101c0f Azure:

[GitHub] [hudi] hudi-bot commented on pull request #3877: [HUDI-2590][WIP] Test different keygen with and without glob path

2021-12-29 Thread GitBox
hudi-bot commented on pull request #3877: URL: https://github.com/apache/hudi/pull/3877#issuecomment-1002837366 ## CI report: * c721ba387780ecac09feb45b79d23c98b9d4d39a Azure:

[GitHub] [hudi] hudi-bot removed a comment on pull request #3877: [HUDI-2590][WIP] Test different keygen with and without glob path

2021-12-29 Thread GitBox
hudi-bot removed a comment on pull request #3877: URL: https://github.com/apache/hudi/pull/3877#issuecomment-961588698 ## CI report: * c721ba387780ecac09feb45b79d23c98b9d4d39a Azure:

[GitHub] [hudi] hudi-bot commented on pull request #4470: [HUDI-3083] Support component data types for flink bulk_insert

2021-12-29 Thread GitBox
hudi-bot commented on pull request #4470: URL: https://github.com/apache/hudi/pull/4470#issuecomment-1002837046 ## CI report: * f718cea8b001eaeb132d97b2ae0e49f1bc101c0f Azure:

[GitHub] [hudi] hudi-bot removed a comment on pull request #4470: [HUDI-3083] Support component data types for flink bulk_insert

2021-12-29 Thread GitBox
hudi-bot removed a comment on pull request #4470: URL: https://github.com/apache/hudi/pull/4470#issuecomment-1002660277 ## CI report: * f718cea8b001eaeb132d97b2ae0e49f1bc101c0f Azure:

[GitHub] [hudi] vinothchandar commented on pull request #3173: [HUDI-1951] Add bucket hash index, compatible with the hive bucket

2021-12-29 Thread GitBox
vinothchandar commented on pull request #3173: URL: https://github.com/apache/hudi/pull/3173#issuecomment-1002832151 @minihippo I was thinking we can name all parameters `hoodie.storage.layout..` instead, but the space curve PRs are all named `hoodie.layout.optimize` anyway. So I think

[GitHub] [hudi] vinothchandar removed a comment on pull request #3173: [HUDI-1951] Add bucket hash index, compatible with the hive bucket

2021-12-29 Thread GitBox
vinothchandar removed a comment on pull request #3173: URL: https://github.com/apache/hudi/pull/3173#issuecomment-1002829514 @minihippo I see you have added a new config class now. Can this be called `HoodieStorageLayoutConfig` and properties all named `hoodie.storage.layout.*` --

[GitHub] [hudi] vinothchandar commented on pull request #3173: [HUDI-1951] Add bucket hash index, compatible with the hive bucket

2021-12-29 Thread GitBox
vinothchandar commented on pull request #3173: URL: https://github.com/apache/hudi/pull/3173#issuecomment-1002829514 @minihippo I see you have added a new config class now. Can this be called `HoodieStorageLayoutConfig` and properties all named `hoodie.storage.layout.*` -- This is an

[GitHub] [hudi] vinothchandar commented on pull request #3173: [HUDI-1951] Add bucket hash index, compatible with the hive bucket

2021-12-29 Thread GitBox
vinothchandar commented on pull request #3173: URL: https://github.com/apache/hudi/pull/3173#issuecomment-1002828445 Looking into the failures. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [hudi] vinothchandar commented on a change in pull request #4472: [MINOR] - Publishing blog for Zorder

2021-12-29 Thread GitBox
vinothchandar commented on a change in pull request #4472: URL: https://github.com/apache/hudi/pull/4472#discussion_r776534284 ## File path: website/blog/2021-12-29-hudi-zorder-and-hilbert-space-filling-curves.md ## @@ -0,0 +1,319 @@ +--- +title: "Hudi ZOrder and Hilbert

[GitHub] [hudi] kywe665 opened a new pull request #4472: [MINOR] - Publishing blog for Zorder

2021-12-29 Thread GitBox
kywe665 opened a new pull request #4472: URL: https://github.com/apache/hudi/pull/4472 ## What is the purpose of the pull request Publishing blog for Zorder ## Brief change log Publishing blog for Zorder and supporting img artifacts ## Verify this pull request

[GitHub] [hudi] vingov edited a comment on issue #4429: [SUPPORT] Spark SQL CTAS command doesn't work with 0.10.0 version and Spark 3.1.1

2021-12-29 Thread GitBox
vingov edited a comment on issue #4429: URL: https://github.com/apache/hudi/issues/4429#issuecomment-1002814514 @xushiyan - Thanks for confirming that it works with 3.1.2, my initial error was on Spark 3.1.1, after your last message, I updated my Spark to 3.1.2, now it works. dbt

[GitHub] [hudi] vingov edited a comment on issue #4429: [SUPPORT] Spark SQL CTAS command doesn't work with 0.10.0 version and Spark 3.1.1

2021-12-29 Thread GitBox
vingov edited a comment on issue #4429: URL: https://github.com/apache/hudi/issues/4429#issuecomment-1002814514 @xushiyan - Thanks for confirming that it works with 3.1.2, my initial error was on Spark 3.1.1, after your last message, I updated my Spark to 3.1.2, even after that update I

[GitHub] [hudi] vingov commented on issue #4429: [SUPPORT] Spark SQL CTAS command doesn't work with 0.10.0 version and Spark 3.1.1

2021-12-29 Thread GitBox
vingov commented on issue #4429: URL: https://github.com/apache/hudi/issues/4429#issuecomment-1002814514 @xushiyan - Thanks for confirming that it works with 3.1.2, my initial error was on Spark 3.1.1, after your last message, I updated my Spark to 3.1.2, even after that update I got a

[GitHub] [hudi] nsivabalan commented on pull request #4405: HUDI-3068 Fixing sync all partitions

2021-12-29 Thread GitBox
nsivabalan commented on pull request #4405: URL: https://github.com/apache/hudi/pull/4405#issuecomment-1002811744 I made a hacky fix in .HMSDDLExecutor.updatePartitionsToTable to check if my theory is right ``` Map params = new HashMap<>(); params.put("numFiles","0");

[GitHub] [hudi] nsivabalan edited a comment on pull request #4405: HUDI-3068 Fixing sync all partitions

2021-12-29 Thread GitBox
nsivabalan edited a comment on pull request #4405: URL: https://github.com/apache/hudi/pull/4405#issuecomment-1002810449 sorry about the long question. could not make it succinct. @bvaradar @codope @vinothchandar @xushiyan : Need some guidance on updating partitions to hive.

[GitHub] [hudi] nsivabalan commented on pull request #4405: HUDI-3068 Fixing sync all partitions

2021-12-29 Thread GitBox
nsivabalan commented on pull request #4405: URL: https://github.com/apache/hudi/pull/4405#issuecomment-1002810449 sorry about the long question. could not make it succinct. @bvaradar @codope @vinothchandar : Need some guidance on updating partitions to hive. [Reference code

[GitHub] [hudi] nsivabalan commented on a change in pull request #4468: [Issue: #2802] Fixing Hive getSchema for RT tables

2021-12-29 Thread GitBox
nsivabalan commented on a change in pull request #4468: URL: https://github.com/apache/hudi/pull/4468#discussion_r776510489 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/AbstractRealtimeRecordReader.java ## @@ -77,19 +74,17 @@ private boolean

[jira] [Updated] (HUDI-3130) Hive read fails when different partitions have different schemas

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3130: -- Fix Version/s: 0.11.0 > Hive read fails when different partitions have different

[jira] [Commented] (HUDI-1965) Add a FAQ around upgrading Hudi version in EMR cluster

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17466607#comment-17466607 ] sivabalan narayanan commented on HUDI-1965: --- [~bhasudha] : Can we close this one too? I guess

[jira] [Resolved] (HUDI-3011) Add support to start incremental consumption from begin time rather than latest commit time with S3EventsHoodieIncrSource

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-3011. --- > Add support to start incremental consumption from begin time rather than > latest

[GitHub] [hudi] nsivabalan commented on issue #4318: [SUPPORT] Duplicate records in COW table within same partition path

2021-12-29 Thread GitBox
nsivabalan commented on issue #4318: URL: https://github.com/apache/hudi/issues/4318#issuecomment-1002791982 @stym06 : Can you give this a try https://github.com/apache/hudi/pull/3222. would help to certify the patch too. -- This is an automated message from the Apache Git Service. To

[GitHub] [hudi] nsivabalan edited a comment on issue #2802: Hive read issues when different partition have different schemas.

2021-12-29 Thread GitBox
nsivabalan edited a comment on issue #2802: URL: https://github.com/apache/hudi/issues/2802#issuecomment-1002788785 @aditiwari01 : sorry, can you point me to the patch which you have put up on this regard. I can take a stab at it. I have filed a jira

[jira] [Updated] (HUDI-3130) Hive read fails when different partitions have different schemas

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3130: -- Description: Hive read fails when different partitions have different schemas  

[jira] [Updated] (HUDI-3130) Hive read fails when different partitions have different schemas

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3130: -- Labels: sev:high (was: ) > Hive read fails when different partitions have different

[jira] [Created] (HUDI-3130) Hive read fails when different partitions have different schemas

2021-12-29 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-3130: - Summary: Hive read fails when different partitions have different schemas Key: HUDI-3130 URL: https://issues.apache.org/jira/browse/HUDI-3130 Project:

[GitHub] [hudi] nsivabalan commented on issue #2802: Hive read issues when different partition have different schemas.

2021-12-29 Thread GitBox
nsivabalan commented on issue #2802: URL: https://github.com/apache/hudi/issues/2802#issuecomment-1002788785 @aditiwari01 : sorry, can you point me to the patch which you have put up on this regard. I can take a stab at it. -- This is an automated message from the Apache Git Service.

[GitHub] [hudi] nsivabalan commented on issue #4177: [SUPPORT] org.apache.hudi.exception.HoodieException: Unknown versionCode:2

2021-12-29 Thread GitBox
nsivabalan commented on issue #4177: URL: https://github.com/apache/hudi/issues/4177#issuecomment-1002788263 thanks @singaretti for the update. glad to know everything is resolved now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[jira] [Updated] (HUDI-3032) Do not clean the log files right after compaction for metadata table

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3032: -- Fix Version/s: 0.11.0 > Do not clean the log files right after compaction for metadata

[jira] [Updated] (HUDI-3032) Do not clean the log files right after compaction for metadata table

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3032: -- Fix Version/s: 0.10.1 > Do not clean the log files right after compaction for metadata

[jira] [Reopened] (HUDI-3070) Improve Test

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reopened HUDI-3070: --- > Improve Test > > > Key: HUDI-3070 > URL:

[jira] [Resolved] (HUDI-3070) Improve Test

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-3070. --- > Improve Test > > > Key: HUDI-3070 > URL:

[jira] [Updated] (HUDI-3052) Flaky TestJsonKafkaSource in CI runs

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3052: -- Fix Version/s: 0.10.1 > Flaky TestJsonKafkaSource in CI runs >

[jira] [Updated] (HUDI-3064) FileSystemBasedLockProviderTestClass - tryLock doesn't honor retries

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3064: -- Fix Version/s: 0.10.1 > FileSystemBasedLockProviderTestClass - tryLock doesn't honor

[jira] [Updated] (HUDI-3054) Fix flaky TestHoodieClientMultiWriter. testHoodieClientBasicMultiWriter

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3054: -- Fix Version/s: 0.10.1 > Fix flaky TestHoodieClientMultiWriter.

[jira] [Created] (HUDI-3129) Test Hudi Subtask

2021-12-29 Thread Vinoth Chandar (Jira)
Vinoth Chandar created HUDI-3129: Summary: Test Hudi Subtask Key: HUDI-3129 URL: https://issues.apache.org/jira/browse/HUDI-3129 Project: Apache Hudi Issue Type: Sub-task

[jira] [Updated] (HUDI-3128) [UMBRELLA] Test Hudi Epic

2021-12-29 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-3128: - Priority: Blocker (was: Major) > [UMBRELLA] Test Hudi Epic > - > >

[jira] [Updated] (HUDI-3128) [UMBRELLA] Test Hudi Epic

2021-12-29 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-3128: - Fix Version/s: 0.11.0 > [UMBRELLA] Test Hudi Epic > - > >

[jira] [Created] (HUDI-3128) [UMBRELLA] Test Hudi Epic

2021-12-29 Thread Vinoth Chandar (Jira)
Vinoth Chandar created HUDI-3128: Summary: [UMBRELLA] Test Hudi Epic Key: HUDI-3128 URL: https://issues.apache.org/jira/browse/HUDI-3128 Project: Apache Hudi Issue Type: New Feature

[jira] [Updated] (HUDI-3043) Test failure with TestHoodieDeltaStreamerWithMultiWriter.testUpsertsContinuousModeWithMultipleWriters

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3043: -- Fix Version/s: 0.10.1 > Test failure with >

[jira] [Updated] (HUDI-2962) Support JVM based local process lock provider implementation

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2962: -- Fix Version/s: 0.10.1 > Support JVM based local process lock provider implementation >

[jira] [Updated] (HUDI-2958) Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert to insert data which contains decimal Type.

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2958: -- Fix Version/s: 0.10.1 > Automatically set spark.sql.parquet.writelegacyformat; When

[jira] [Updated] (HUDI-3001) clean up temp marker directory when finish bootstrap operation.

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3001: -- Fix Version/s: 0.10.1 > clean up temp marker directory when finish bootstrap

[jira] [Updated] (HUDI-3028) Spark binary download sometimes takes a long time in Azure CI IT tests

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3028: -- Fix Version/s: 0.10.1 > Spark binary download sometimes takes a long time in Azure CI

[jira] [Updated] (HUDI-3025) Integ tests are failing in azure CI with namenode going to safe mode

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3025: -- Fix Version/s: 0.10.1 > Integ tests are failing in azure CI with namenode going to safe

[jira] [Updated] (HUDI-2100) [UMBRELLA] Support Space curve for hudi

2021-12-29 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2100: - Fix Version/s: 0.11.0 > [UMBRELLA] Support Space curve for hudi >

[jira] [Updated] (HUDI-2946) Upgrade maven plugin to make Hudi be compatible with higher Java versions

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2946: -- Fix Version/s: 0.11.0 0.10.1 > Upgrade maven plugin to make Hudi be

[jira] [Updated] (HUDI-2938) Code Refactor: Metadata util to get latest file slices for readers and writers

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2938: -- Fix Version/s: 0.10.1 > Code Refactor: Metadata util to get latest file slices for

[jira] [Updated] (HUDI-2985) Shade jackson for hudi flink bundle jar

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2985: -- Fix Version/s: 0.10.1 > Shade jackson for hudi flink bundle jar >

[jira] [Updated] (HUDI-431) Support Parquet in MOR log files

2021-12-29 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-431: Reviewers: Vinoth Chandar > Support Parquet in MOR log files > > >

[jira] [Updated] (HUDI-2974) Make the prefix for metrics name configurable

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2974: -- Fix Version/s: 0.11.0 0.10.1 > Make the prefix for metrics name

[jira] [Updated] (HUDI-2892) Pending Clustering may stain the ActiveTimeLine and lead to incomplete query results

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2892: -- Fix Version/s: 0.10.1 > Pending Clustering may stain the ActiveTimeLine and lead to

[jira] [Updated] (HUDI-2952) Metadata table compaction fails for non partitioned dataset

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2952: -- Fix Version/s: 0.10.1 > Metadata table compaction fails for non partitioned dataset >

[jira] [Updated] (HUDI-2849) Improve job/stage description in spark UI for hudi write

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2849: -- Fix Version/s: 0.11.0 0.10.1 > Improve job/stage description in

[jira] [Updated] (HUDI-2901) Fixed the bug clustering jobs are not running in parallel

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2901: -- Fix Version/s: 0.10.1 > Fixed the bug clustering jobs are not running in parallel >

[jira] [Updated] (HUDI-2966) Add TaskCompletionListener for HoodieMergeOnReadRDD to close logScanner

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2966: -- Fix Version/s: 0.10.1 > Add TaskCompletionListener for HoodieMergeOnReadRDD to close

[jira] [Updated] (HUDI-2779) Cache BaseDir if HudiTableNotFound Exception thrown

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2779: -- Fix Version/s: 0.10.1 > Cache BaseDir if HudiTableNotFound Exception thrown >

[jira] [Updated] (HUDI-2665) Overflow of DataOutputStream may lead to corrupted log block

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2665: -- Fix Version/s: 0.10.1 > Overflow of DataOutputStream may lead to corrupted log block >

[jira] [Updated] (HUDI-2942) HoodieCombineHiveInputFormat need add error info in hive integration

2021-12-29 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2942: -- Fix Version/s: 0.10.1 > HoodieCombineHiveInputFormat need add error info in hive

<    1   2   3   >