[GitHub] [hudi] yuzhaojing commented on pull request #3888: [HUDI-2624] Implement Non Index type for HUDI

2021-11-02 Thread GitBox
yuzhaojing commented on pull request #3888: URL: https://github.com/apache/hudi/pull/3888#issuecomment-958673059 > @yuzhaojing check CI? CI is all successed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[jira] [Updated] (HUDI-2616) Implement BloomIndex for Dataset

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2616: - Priority: Critical (was: Blocker) > Implement BloomIndex for Dataset >

[jira] [Updated] (HUDI-2616) Implement BloomIndex for Dataset

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2616: - Fix Version/s: (was: 0.10.0) > Implement BloomIndex for Dataset >

[jira] [Updated] (HUDI-2531) [UMBRELLA] Support Dataset APIs in writer paths

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2531: - Fix Version/s: (was: 0.10.0) > [UMBRELLA] Support Dataset APIs in writer paths >

[jira] [Updated] (HUDI-2620) Benchmark SparkDataFrameWriteClient

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2620: - Fix Version/s: (was: 0.10.0) > Benchmark SparkDataFrameWriteClient >

[jira] [Updated] (HUDI-2620) Benchmark SparkDataFrameWriteClient

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2620: - Priority: Major (was: Blocker) > Benchmark SparkDataFrameWriteClient >

[jira] [Updated] (HUDI-2665) Overflow of DataOutputStream may lead to corrupted log block

2021-11-02 Thread ZiyueGuan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan updated HUDI-2665: Priority: Minor (was: Major) > Overflow of DataOutputStream may lead to corrupted log block >

[jira] [Updated] (HUDI-1430) Implement SparkDataFrameWriteClient with SimpleIndex

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-1430: - Fix Version/s: (was: 0.10.0) > Implement SparkDataFrameWriteClient with SimpleIndex >

[jira] [Updated] (HUDI-2615) Decouple HoodieRecordPayload with Hoodie table, table services, and index

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2615: - Fix Version/s: (was: 0.10.0) > Decouple HoodieRecordPayload with Hoodie table, table services, and

[jira] [Updated] (HUDI-2615) Decouple HoodieRecordPayload with Hoodie table, table services, and index

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2615: - Priority: Major (was: Blocker) > Decouple HoodieRecordPayload with Hoodie table, table services, and

[jira] [Updated] (HUDI-1430) Implement SparkDataFrameWriteClient with SimpleIndex

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-1430: - Priority: Major (was: Blocker) > Implement SparkDataFrameWriteClient with SimpleIndex >

[jira] [Updated] (HUDI-2621) Enhance DataFrameWriter with small file handling

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2621: - Priority: Major (was: Blocker) > Enhance DataFrameWriter with small file handling >

[jira] [Updated] (HUDI-2617) Implement HBase Index for Dataset

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2617: - Priority: Major (was: Blocker) > Implement HBase Index for Dataset >

[jira] [Updated] (HUDI-2618) Implement write operations other than upsert in SparkDataFrameWriteClient

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2618: - Priority: Major (was: Blocker) > Implement write operations other than upsert in

[jira] [Updated] (HUDI-2617) Implement HBase Index for Dataset

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2617: - Fix Version/s: (was: 0.10.0) > Implement HBase Index for Dataset >

[jira] [Updated] (HUDI-2621) Enhance DataFrameWriter with small file handling

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2621: - Fix Version/s: (was: 0.10.0) > Enhance DataFrameWriter with small file handling >

[jira] [Updated] (HUDI-2622) Enhance DataFrameWriter with LazyIterator and SpillableMap

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2622: - Fix Version/s: (was: 0.10.0) > Enhance DataFrameWriter with LazyIterator and SpillableMap >

[jira] [Updated] (HUDI-2618) Implement write operations other than upsert in SparkDataFrameWriteClient

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2618: - Fix Version/s: (was: 0.10.0) > Implement write operations other than upsert in

[jira] [Updated] (HUDI-2622) Enhance DataFrameWriter with LazyIterator and SpillableMap

2021-11-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-2622: - Priority: Major (was: Blocker) > Enhance DataFrameWriter with LazyIterator and SpillableMap >

[jira] [Resolved] (HUDI-233) Redo log statements using SLF4J

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit resolved HUDI-233. -- Resolution: Fixed > Redo log statements using SLF4J > > >

[jira] [Updated] (HUDI-1872) Move HoodieFlinkStreamer into hudi-utilities module

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-1872: -- Status: Patch Available (was: In Progress) > Move HoodieFlinkStreamer into hudi-utilities module >

[jira] [Commented] (HUDI-1528) hudi-sync-tools error

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437738#comment-17437738 ] Sagar Sumit commented on HUDI-1528: --- It's working now. I have shared code snippet here

[jira] [Closed] (HUDI-1528) hudi-sync-tools error

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-1528. - Resolution: Fixed > hudi-sync-tools error > - > > Key: HUDI-1528 >

[GitHub] [hudi] xushiyan commented on pull request #3053: [HUDI-1932] Update Hive sync timestamp when change detected

2021-11-02 Thread GitBox
xushiyan commented on pull request #3053: URL: https://github.com/apache/hudi/pull/3053#issuecomment-958666489 @zuyanton yes we're planning it for 0.10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[jira] [Updated] (HUDI-2663) Incorrect deletion of heartbeat files for inflight commits

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2663: - Priority: Blocker (was: Critical) > Incorrect deletion of heartbeat files for inflight commits >

[jira] [Updated] (HUDI-2663) Incorrect deletion of heartbeat files for inflight commits

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2663: - Story Points: 10 > Incorrect deletion of heartbeat files for inflight commits >

[jira] [Assigned] (HUDI-2663) Incorrect deletion of heartbeat files for inflight commits

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-2663: Assignee: Vinoth Chandar > Incorrect deletion of heartbeat files for inflight commits >

[GitHub] [hudi] hudi-bot edited a comment on pull request #3911: [HUDI-2676] Hudi should synchronize owner information to hudi _rt/_ro…

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3911: URL: https://github.com/apache/hudi/pull/3911#issuecomment-958636680 ## CI report: * 90b58a3afad964af9d252a3633b555a21253df7d Azure:

[jira] [Updated] (HUDI-1475) Fix documentation of preCombine to clarify when this API is used by Hudi

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-1475: -- Status: Closed (was: Patch Available) > Fix documentation of preCombine to clarify when this API is

[jira] [Updated] (HUDI-718) java.lang.ClassCastException during upsert

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-718: - Status: In Progress (was: Open) > java.lang.ClassCastException during upsert >

[jira] [Updated] (HUDI-718) java.lang.ClassCastException during upsert

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-718: - Status: Closed (was: Patch Available) > java.lang.ClassCastException during upsert >

[jira] [Updated] (HUDI-718) java.lang.ClassCastException during upsert

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-718: - Status: Patch Available (was: In Progress) > java.lang.ClassCastException during upsert >

[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3486: URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684 ## CI report: * d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN * 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN *

[jira] [Commented] (HUDI-943) Slow performance observed when inserting data into Hudi table

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437732#comment-17437732 ] Sagar Sumit commented on HUDI-943: -- [~h117561964] [~vbalaji] Is thi still an issue? HoodieSparkSqlWriter

[GitHub] [hudi] hudi-bot edited a comment on pull request #3614: [HUDI-2370] Supports data encryption

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3614: URL: https://github.com/apache/hudi/pull/3614#issuecomment-914114290 ## CI report: * a3677e66a1fb13c1a91d6beb977b00ddfdd6a51e Azure:

[jira] [Assigned] (HUDI-2151) Make performant out-of-box configs

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-2151: Assignee: Raymond Xu (was: sivabalan narayanan) > Make performant out-of-box configs >

[GitHub] [hudi] hudi-bot edited a comment on pull request #3899: [HUDI-2660] Delete the view storage properties first before creation

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3899: URL: https://github.com/apache/hudi/pull/3899#issuecomment-956165515 ## CI report: * c30db533861087c73d6d71e68cc6fdc00985803b Azure:

[jira] [Commented] (HUDI-1609) Issues w/ using hive metastore by disabling jdbc

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437728#comment-17437728 ] Sagar Sumit commented on HUDI-1609: --- It should have been fixed now. I'll verify it. > Issues w/ using

[jira] [Updated] (HUDI-1609) Issues w/ using hive metastore by disabling jdbc

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-1609: -- Status: In Progress (was: Open) > Issues w/ using hive metastore by disabling jdbc >

[jira] [Updated] (HUDI-2480) FileSlice after pending compaction-requested instant-time is ignored by MOR snapshot reader

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-2480: -- Status: Patch Available (was: In Progress) > FileSlice after pending compaction-requested instant-time

[jira] [Updated] (HUDI-2480) FileSlice after pending compaction-requested instant-time is ignored by MOR snapshot reader

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-2480: -- Status: In Progress (was: Open) > FileSlice after pending compaction-requested instant-time is ignored

[jira] [Commented] (HUDI-2493) Verify removing glob pattern works w/ all key generators

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437727#comment-17437727 ] Sagar Sumit commented on HUDI-2493: --- [~rxu][~shivnarayan]Can we close this in favour of

[GitHub] [hudi] vinothchandar merged pull request #3907: [HUDI-2670] - relative links broken in docs

2021-11-02 Thread GitBox
vinothchandar merged pull request #3907: URL: https://github.com/apache/hudi/pull/3907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[hudi] branch asf-site updated: [HUDI-2670] - relative links broken in docs (#3907)

2021-11-02 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new aee584d [HUDI-2670] - relative links broken

[GitHub] [hudi] vinothchandar commented on pull request #3907: [HUDI-2670] - relative links broken in docs

2021-11-02 Thread GitBox
vinothchandar commented on pull request #3907: URL: https://github.com/apache/hudi/pull/3907#issuecomment-958653666 @kywe665 there are 24 commits in this PR, even though there are only 3 files changed? For every PR, you can use a new branch and rebase that to asf-site prior? it can avoid

[jira] [Updated] (HUDI-2670) Fix broken relative links

2021-11-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2670: - Labels: pull-request-available (was: ) > Fix broken relative links > - >

[GitHub] [hudi] vinothchandar commented on pull request #3907: [HUDI-2670] - relative links broken in docs

2021-11-02 Thread GitBox
vinothchandar commented on pull request #3907: URL: https://github.com/apache/hudi/pull/3907#issuecomment-958653341 Seems to build locally. landing to make asf-site green again. probably need a better soln? cc @vingov -- This is an automated message from the Apache Git Service.

[jira] [Updated] (HUDI-2509) OverwriteNonDefaultsWithLatestAvroPayload doesn`t work when upsert data with some null value column

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-2509: -- Status: Patch Available (was: In Progress) > OverwriteNonDefaultsWithLatestAvroPayload doesn`t work

[jira] [Updated] (HUDI-2509) OverwriteNonDefaultsWithLatestAvroPayload doesn`t work when upsert data with some null value column

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-2509: -- Status: In Progress (was: Open) > OverwriteNonDefaultsWithLatestAvroPayload doesn`t work when upsert

[jira] [Updated] (HUDI-1976) Upgrade hive, jackson, log4j, hadoop to remove vulnerability

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-1976: -- Status: Patch Available (was: In Progress) > Upgrade hive, jackson, log4j, hadoop to remove

[jira] [Updated] (HUDI-1975) Upgrade java-prometheus-client from 3.1.2 to 4.x

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-1975: -- Status: Patch Available (was: In Progress) > Upgrade java-prometheus-client from 3.1.2 to 4.x >

[jira] [Updated] (HUDI-1864) Support for java.time.LocalDate in TimestampBasedAvroKeyGenerator

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-1864: -- Status: Patch Available (was: In Progress) > Support for java.time.LocalDate in

[jira] [Updated] (HUDI-83) Map Timestamp type in spark to corresponding Timestamp type in Hive during Hive sync

2021-11-02 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-83?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-83: Status: Patch Available (was: In Progress) > Map Timestamp type in spark to corresponding Timestamp type in

[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3486: URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684 ## CI report: * d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN * 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN *

[GitHub] [hudi] hudi-bot edited a comment on pull request #3888: [HUDI-2624] Implement Non Index type for HUDI

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3888: URL: https://github.com/apache/hudi/pull/3888#issuecomment-954503596 ## CI report: * 0bb6cf636d6a4e9e902706a28364845a7609e38d Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3888: [HUDI-2624] Implement Non Index type for HUDI

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3888: URL: https://github.com/apache/hudi/pull/3888#issuecomment-954503596 ## CI report: * 0bb6cf636d6a4e9e902706a28364845a7609e38d Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3486: URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684 ## CI report: * d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN * 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN *

[GitHub] [hudi] prashantwason commented on pull request #3871: [HUDI-2593][WIP] Enabling virtual keys for the metadata table

2021-11-02 Thread GitBox
prashantwason commented on pull request #3871: URL: https://github.com/apache/hudi/pull/3871#issuecomment-958638352 But partition path for the metadata table are hardcoded. Can that be helpful? Removing the fields will save a lot of storage space from record level index. -- This is an

[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock

2021-11-02 Thread GitBox
zhedoubushishi commented on a change in pull request #3486: URL: https://github.com/apache/hudi/pull/3486#discussion_r741605659 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/DynamoDBBasedLockProvider.java ## @@ -0,0 +1,226

[GitHub] [hudi] hudi-bot edited a comment on pull request #3899: [HUDI-2660] Delete the view storage properties first before creation

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3899: URL: https://github.com/apache/hudi/pull/3899#issuecomment-956165515 ## CI report: * 245ea82852227fb3bd29aa389a64ec4f291afb0f Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3911: [HUDI-2676] Hudi should synchronize owner information to hudi _rt/_ro…

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3911: URL: https://github.com/apache/hudi/pull/3911#issuecomment-958636680 ## CI report: * 90b58a3afad964af9d252a3633b555a21253df7d Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3614: [HUDI-2370] Supports data encryption

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3614: URL: https://github.com/apache/hudi/pull/3614#issuecomment-914114290 ## CI report: * 61156c4e958c1b20c3479a55ef71f2e11891398a Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3486: [HUDI-2314] Add support for DynamoDb based lock

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3486: URL: https://github.com/apache/hudi/pull/3486#issuecomment-899911684 ## CI report: * d2b00796c9564088aa8533431c73251993f688d4 UNKNOWN * 99853468aec1becd1112c0ffba6ccf5f604e713d UNKNOWN *

[GitHub] [hudi] hudi-bot commented on pull request #3911: [HUDI-2676] Hudi should synchronize owner information to hudi _rt/_ro…

2021-11-02 Thread GitBox
hudi-bot commented on pull request #3911: URL: https://github.com/apache/hudi/pull/3911#issuecomment-958636680 ## CI report: * 90b58a3afad964af9d252a3633b555a21253df7d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis`

[GitHub] [hudi] hudi-bot edited a comment on pull request #3899: [HUDI-2660] Delete the view storage properties first before creation

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3899: URL: https://github.com/apache/hudi/pull/3899#issuecomment-956165515 ## CI report: * 245ea82852227fb3bd29aa389a64ec4f291afb0f Azure:

[GitHub] [hudi] zhedoubushishi commented on a change in pull request #3486: [HUDI-2314] Add support for DynamoDb based lock

2021-11-02 Thread GitBox
zhedoubushishi commented on a change in pull request #3486: URL: https://github.com/apache/hudi/pull/3486#discussion_r741604415 ## File path: hudi-client/hudi-client-common/pom.xml ## @@ -218,6 +222,27 @@ ${zk-curator.version} test + +

[GitHub] [hudi] hudi-bot edited a comment on pull request #3614: [HUDI-2370] Supports data encryption

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3614: URL: https://github.com/apache/hudi/pull/3614#issuecomment-914114290 ## CI report: * 61156c4e958c1b20c3479a55ef71f2e11891398a Azure:

[GitHub] [hudi] xiarixiaoyao commented on pull request #2907: [HUDI-1873] collect() call causing issues with very large upserts

2021-11-02 Thread GitBox
xiarixiaoyao commented on pull request #2907: URL: https://github.com/apache/hudi/pull/2907#issuecomment-958635857 @vinothchandar @nsivabalan @mpouttu hello, this change cause performance degradation in our test env。rdd.isEmpty will trigger a partition level calculation, then in

[jira] [Resolved] (HUDI-1721) run_sync_tool support hive3.1.2 on hadoop3.1.4

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar resolved HUDI-1721. -- Resolution: Won't Fix > run_sync_tool support hive3.1.2 on hadoop3.1.4 >

[jira] [Updated] (HUDI-1721) run_sync_tool support hive3.1.2 on hadoop3.1.4

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1721: - Status: Closed (was: Patch Available) > run_sync_tool support hive3.1.2 on hadoop3.1.4 >

[jira] [Reopened] (HUDI-1721) run_sync_tool support hive3.1.2 on hadoop3.1.4

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reopened HUDI-1721: -- > run_sync_tool support hive3.1.2 on hadoop3.1.4 > ---

[jira] [Resolved] (HUDI-2048) HoodieRealtimeInputFormatUtils#groupLogsByBaseFile throws NPE for file group that has only logs

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar resolved HUDI-2048. -- Resolution: Fixed > HoodieRealtimeInputFormatUtils#groupLogsByBaseFile throws NPE for file

[jira] [Reopened] (HUDI-2048) HoodieRealtimeInputFormatUtils#groupLogsByBaseFile throws NPE for file group that has only logs

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reopened HUDI-2048: -- > HoodieRealtimeInputFormatUtils#groupLogsByBaseFile throws NPE for file group > that has only

[jira] [Updated] (HUDI-2048) HoodieRealtimeInputFormatUtils#groupLogsByBaseFile throws NPE for file group that has only logs

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2048: - Status: Closed (was: Patch Available) > HoodieRealtimeInputFormatUtils#groupLogsByBaseFile

[jira] [Updated] (HUDI-2637) Triage all bugs around Multi-writer and certify the tested flows

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2637: - Story Points: 20 > Triage all bugs around Multi-writer and certify the tested flows >

[jira] [Assigned] (HUDI-2637) Triage all bugs around Multi-writer and certify the tested flows

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-2637: Assignee: Vinoth Chandar (was: sivabalan narayanan) > Triage all bugs around Multi-writer

[GitHub] [hudi] xiarixiaoyao commented on pull request #3911: [HUDI-2676] Hudi should synchronize owner information to hudi _rt/_ro…

2021-11-02 Thread GitBox
xiarixiaoyao commented on pull request #3911: URL: https://github.com/apache/hudi/pull/3911#issuecomment-958628416 @leesf @nsivabalan could you pls help me review this code thanks. a minor fix, we need synchronize owner information to hudi _rt/_ro table. -- This is an automated

[GitHub] [hudi] nsivabalan commented on pull request #3416: [HUDI-2362] Add external config file support

2021-11-02 Thread GitBox
nsivabalan commented on pull request #3416: URL: https://github.com/apache/hudi/pull/3416#issuecomment-958627856 @xushiyan : since you have already reviewed this, I will let you drive this home. -- This is an automated message from the Apache Git Service. To respond to the message,

[jira] [Updated] (HUDI-2634) Improve bootstrap performance for very large tables

2021-11-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2634: -- Status: In Progress (was: Open) > Improve bootstrap performance for very large tables

[jira] [Updated] (HUDI-2634) Improve bootstrap performance for very large tables

2021-11-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2634: -- Status: Patch Available (was: In Progress) > Improve bootstrap performance for very

[jira] [Updated] (HUDI-2591) Double bootstrap of metadata table when upgrade is involved

2021-11-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2591: -- Status: In Progress (was: Open) > Double bootstrap of metadata table when upgrade is

[jira] [Updated] (HUDI-2591) Double bootstrap of metadata table when upgrade is involved

2021-11-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2591: -- Status: Patch Available (was: In Progress) > Double bootstrap of metadata table when

[jira] [Updated] (HUDI-2675) Not an Avro data file

2021-11-02 Thread Jira
[ https://issues.apache.org/jira/browse/HUDI-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 董可伦 updated HUDI-2675: -- Description: There are three places where I have encountered this exception,I'm not sure if there are other places

[jira] [Updated] (HUDI-2676) Hudi should synchronize owner information to hudi _rt/_ro table。

2021-11-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2676: - Labels: pull-request-available (was: ) > Hudi should synchronize owner information to hudi

[jira] [Updated] (HUDI-2675) Not an Avro data file

2021-11-02 Thread Jira
[ https://issues.apache.org/jira/browse/HUDI-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 董可伦 updated HUDI-2675: -- Description: There are three places where I have encountered this exception,I'm not sure if there are other places

[GitHub] [hudi] xiarixiaoyao opened a new pull request #3911: [HUDI-2676] Hudi should synchronize owner information to hudi _rt/_ro…

2021-11-02 Thread GitBox
xiarixiaoyao opened a new pull request #3911: URL: https://github.com/apache/hudi/pull/3911 … table. ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*

[jira] [Assigned] (HUDI-2602) Publish design doc/RFC for metadata based range index

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-2602: Assignee: sivabalan narayanan > Publish design doc/RFC for metadata based range index >

[GitHub] [hudi] nsivabalan commented on pull request #3910: [HUDI-2674] hudi hive reader should not print read values.

2021-11-02 Thread GitBox
nsivabalan commented on pull request #3910: URL: https://github.com/apache/hudi/pull/3910#issuecomment-958618774 thanks for catching this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[hudi] branch master updated (b12a25b -> 5517d29)

2021-11-02 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from b12a25b [MINOR] Fixed RAT config for "hudi-utilities-bundle" to ignore transient build-bound artifiacts (#3909)

[GitHub] [hudi] nsivabalan merged pull request #3910: [HUDI-2674] hudi hive reader should not print read values.

2021-11-02 Thread GitBox
nsivabalan merged pull request #3910: URL: https://github.com/apache/hudi/pull/3910 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Updated] (HUDI-2641) One inflight commit rolling back other concurrent inflight commits causing them to fail

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2641: - Parent: HUDI-1456 Issue Type: Sub-task (was: Bug) > One inflight commit rolling back

[jira] [Created] (HUDI-2676) Hudi should synchronize owner information to hudi _rt/_ro table。

2021-11-02 Thread tao meng (Jira)
tao meng created HUDI-2676: -- Summary: Hudi should synchronize owner information to hudi _rt/_ro table。 Key: HUDI-2676 URL: https://issues.apache.org/jira/browse/HUDI-2676 Project: Apache Hudi

[jira] [Updated] (HUDI-2438) [Umbrella] [RFC-34] Implement BigQuerySyncTool for BigQuery Sync

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2438: - Fix Version/s: (was: 0.10.0) 0.11.0 > [Umbrella] [RFC-34] Implement

[GitHub] [hudi] nsivabalan commented on a change in pull request #3910: [HUDI-2674] hudi hive reader should not print read values.

2021-11-02 Thread GitBox
nsivabalan commented on a change in pull request #3910: URL: https://github.com/apache/hudi/pull/3910#discussion_r741592164 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieCombineRealtimeRecordReader.java ## @@ -66,8 +65,6 @@ public

[GitHub] [hudi] hudi-bot edited a comment on pull request #3903: [HUDI-2651] Sync all the missing sql options for HoodieFlinkStreamer

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3903: URL: https://github.com/apache/hudi/pull/3903#issuecomment-957173931 ## CI report: * 9754611552f4db38f7679f54dfe86a3191bb7473 Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3888: [HUDI-2624] Implement Non Index type for HUDI

2021-11-02 Thread GitBox
hudi-bot edited a comment on pull request #3888: URL: https://github.com/apache/hudi/pull/3888#issuecomment-954503596 ## CI report: * 0bb6cf636d6a4e9e902706a28364845a7609e38d Azure:

[jira] [Updated] (HUDI-2438) [Umbrella] [RFC-34] Implement BigQuerySyncTool for BigQuery Sync

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2438: - Priority: Major (was: Blocker) > [Umbrella] [RFC-34] Implement BigQuerySyncTool for BigQuery

[jira] [Updated] (HUDI-2303) TestMereIntoLogOnlyTable with metadata enabled surfaces likely bug

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-2303: - Parent: HUDI-1292 Issue Type: Sub-task (was: Bug) > TestMereIntoLogOnlyTable with

[hudi] branch master updated (6351e5f -> b12a25b)

2021-11-02 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 6351e5f [HUDI-2538] persist some configs to hoodie.properties when the first write (#3823) add b12a25b

[jira] [Resolved] (HUDI-1869) Upgrading Spark3 To 3.1

2021-11-02 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar resolved HUDI-1869. -- Resolution: Fixed > Upgrading Spark3 To 3.1 > --- > > Key:

  1   2   3   4   5   6   7   >