[GitHub] [hudi] hudi-bot commented on pull request #9408: [HUDI-6671] Support 'alter table add partition' sql

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9408: URL: https://github.com/apache/hudi/pull/9408#issuecomment-1672596213 ## CI report: * 89c387c5edc9044786899bc1288e35121df600f9 Azure:

[GitHub] [hudi] zlinsc commented on issue #9319: [SUPPORT] how to use HiveSyncConfig instead of hive configs in DataSourceWriteOptions object

2023-08-09 Thread via GitHub
zlinsc commented on issue #9319: URL: https://github.com/apache/hudi/issues/9319#issuecomment-1672579627 > @zlinsc You can use META_SYNC_DATABASE_NAME and META_SYNC_TABLE_NAME from HoodieSyncConfig. Whether HoodieSyncConfig will replace all the configs in the future? I found that it

[GitHub] [hudi] hudi-bot commented on pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9412: URL: https://github.com/apache/hudi/pull/9412#issuecomment-1672559947 ## CI report: * 24c43e61e9a304224df2ca5e2001551974348671 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9412: URL: https://github.com/apache/hudi/pull/9412#issuecomment-1672553728 ## CI report: * 24c43e61e9a304224df2ca5e2001551974348671 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] danny0405 commented on a diff in pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike

2023-08-09 Thread via GitHub
danny0405 commented on code in PR #9412: URL: https://github.com/apache/hudi/pull/9412#discussion_r1289540657 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableLikeCommand.scala: ## @@ -0,0 +1,112 @@ +/* + * Licensed to

[GitHub] [hudi] hudi-bot commented on pull request #9407: asyncService log prompt incomplete

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9407: URL: https://github.com/apache/hudi/pull/9407#issuecomment-1672526657 ## CI report: * ce0c6dd5877e222dd64ce5ac6434d81168c08727 Azure:

[jira] [Updated] (HUDI-6676) Add command for CreateHoodieTableLike

2023-08-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6676: - Labels: pull-request-available (was: ) > Add command for CreateHoodieTableLike >

[GitHub] [hudi] boneanxs opened a new pull request, #9412: [HUDI-6676] Add command for CreateHoodieTableLike

2023-08-09 Thread via GitHub
boneanxs opened a new pull request, #9412: URL: https://github.com/apache/hudi/pull/9412 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ 1. Create table from non-hudi table 2. Create table from hudi table(The properties related

[GitHub] [hudi] wecharyu commented on a diff in pull request #9408: [HUDI-6671] Support 'alter table add partition' sql

2023-08-09 Thread via GitHub
wecharyu commented on code in PR #9408: URL: https://github.com/apache/hudi/pull/9408#discussion_r1289521158 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/HoodieSqlCommonUtils.scala: ## @@ -330,23 +330,36 @@ object HoodieSqlCommonUtils

[GitHub] [hudi] hudi-bot commented on pull request #9407: asyncService log prompt incomplete

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9407: URL: https://github.com/apache/hudi/pull/9407#issuecomment-1672520993 ## CI report: * ce0c6dd5877e222dd64ce5ac6434d81168c08727 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9411: [HUDI-6674] Add rollback info from metadata table in timeline commands

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9411: URL: https://github.com/apache/hudi/pull/9411#issuecomment-1672514880 ## CI report: * 6a8aa88016ab8c2b2cab779f45ac2ecd409f3742 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9408: [HUDI-6671] Support 'alter table add partition' sql

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9408: URL: https://github.com/apache/hudi/pull/9408#issuecomment-1672514851 ## CI report: * 65e9f9828da86e4558b1830493ead64366e69fae Azure:

[jira] [Updated] (HUDI-6676) Add command for CreateHoodieTableLike

2023-08-09 Thread Hui An (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui An updated HUDI-6676: - Description: 1. Create table from non-hudi table 2. Create table from hudi table(The properties related to Hudi

[GitHub] [hudi] danny0405 commented on a diff in pull request #9407: asyncService log prompt incomplete

2023-08-09 Thread via GitHub
danny0405 commented on code in PR #9407: URL: https://github.com/apache/hudi/pull/9407#discussion_r1289510346 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncService.java: ## @@ -196,11 +196,11 @@ public void

[GitHub] [hudi] danny0405 commented on a diff in pull request #9407: asyncService log prompt incomplete

2023-08-09 Thread via GitHub
danny0405 commented on code in PR #9407: URL: https://github.com/apache/hudi/pull/9407#discussion_r1289510239 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncService.java: ## @@ -196,11 +196,11 @@ public void

[jira] [Created] (HUDI-6676) Add command for CreateHoodieTableLike

2023-08-09 Thread Hui An (Jira)
Hui An created HUDI-6676: Summary: Add command for CreateHoodieTableLike Key: HUDI-6676 URL: https://issues.apache.org/jira/browse/HUDI-6676 Project: Apache Hudi Issue Type: Improvement

[GitHub] [hudi] danny0405 commented on a diff in pull request #9403: Added kafka key as part of hudi metadata columns for Json & Avro KafkaSource

2023-08-09 Thread via GitHub
danny0405 commented on code in PR #9403: URL: https://github.com/apache/hudi/pull/9403#discussion_r1289505376 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/KafkaOffsetPostProcessor.java: ## @@ -54,21 +56,23 @@ public static boolean

[GitHub] [hudi] danny0405 commented on a diff in pull request #9403: Added kafka key as part of hudi metadata columns for Json & Avro KafkaSource

2023-08-09 Thread via GitHub
danny0405 commented on code in PR #9403: URL: https://github.com/apache/hudi/pull/9403#discussion_r1289505153 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/AvroConvertor.java: ## @@ -175,9 +176,12 @@ public GenericRecord

[GitHub] [hudi] danny0405 commented on a diff in pull request #9408: [HUDI-6671] Support 'alter table add partition' sql

2023-08-09 Thread via GitHub
danny0405 commented on code in PR #9408: URL: https://github.com/apache/hudi/pull/9408#discussion_r1289497305 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/HoodieSqlCommonUtils.scala: ## @@ -330,23 +330,36 @@ object HoodieSqlCommonUtils

[GitHub] [hudi] Zouxxyy commented on pull request #4974: [HUDI-3494] Consider triggering condition of MOR compaction during archival

2023-08-09 Thread via GitHub
Zouxxyy commented on PR #4974: URL: https://github.com/apache/hudi/pull/4974#issuecomment-1672490060 - The default triggering condition is the number of delta commits, with the config of hoodie.compact.inline.max.delta.commits. If this setting is larger than the archival config of

[GitHub] [hudi] hudi-bot commented on pull request #9408: [HUDI-6671] Support 'alter table add partition' sql

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9408: URL: https://github.com/apache/hudi/pull/9408#issuecomment-1672487101 ## CI report: * 65e9f9828da86e4558b1830493ead64366e69fae Azure:

[jira] [Updated] (HUDI-6675) Clean action will delete the whole table

2023-08-09 Thread sanqingleo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sanqingleo updated HUDI-6675: - Summary: Clean action will delete the whole table (was: InsertOverwrite will delete the whole table) >

[jira] [Reopened] (HUDI-6675) InsertOverwrite will delete the whole table

2023-08-09 Thread sanqingleo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sanqingleo reopened HUDI-6675: -- > InsertOverwrite will delete the whole table > --- > >

[jira] [Resolved] (HUDI-6675) InsertOverwrite will delete the whole table

2023-08-09 Thread sanqingleo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sanqingleo resolved HUDI-6675. -- > InsertOverwrite will delete the whole table > --- > >

[jira] [Created] (HUDI-6675) InsertOverwrite will delete the whole table

2023-08-09 Thread sanqingleo (Jira)
sanqingleo created HUDI-6675: Summary: InsertOverwrite will delete the whole table Key: HUDI-6675 URL: https://issues.apache.org/jira/browse/HUDI-6675 Project: Apache Hudi Issue Type: Bug

[GitHub] [hudi] someguyLi commented on issue #9363: [SUPPORT] Streaming query loss delete data

2023-08-09 Thread via GitHub
someguyLi commented on issue #9363: URL: https://github.com/apache/hudi/issues/9363#issuecomment-1672439581 > The Hudi table is used like a message queue, so TTL is a general solution for keepping the records aliveness. There is no good solution for this, for Kafka, they throws exception

[GitHub] [hudi] hudi-bot commented on pull request #9411: [HUDI-6674] Add rollback info from metadata table in timeline commands

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9411: URL: https://github.com/apache/hudi/pull/9411#issuecomment-1672423674 ## CI report: * 6a8aa88016ab8c2b2cab779f45ac2ecd409f3742 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8327: [HUDI-5361] Propagate all hoodie configs from spark sqlconf, but don't overwrite values already set

2023-08-09 Thread via GitHub
hudi-bot commented on PR #8327: URL: https://github.com/apache/hudi/pull/8327#issuecomment-1672385859 ## CI report: * 94e4c2e74c6170ceee8c303f7237bd10f2cd334f Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9410: [HUDI-6673] Fix Incremental Query Syntax - Spark SQL Core Flow Test

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9410: URL: https://github.com/apache/hudi/pull/9410#issuecomment-1672381897 ## CI report: * a3bd3418eccb373f200139996d34b8cc71913a62 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9409: [HUDI-6663] New Parquet File Format remove broadcast to fix performance issue for complex file slices

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9409: URL: https://github.com/apache/hudi/pull/9409#issuecomment-1672296128 ## CI report: * d567d80ea610ed8eca248901d310bd40ae4bf8e5 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9411: [HUDI-6674] Add rollback info from metadata table in timeline commands

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9411: URL: https://github.com/apache/hudi/pull/9411#issuecomment-1672282598 ## CI report: * 6a8aa88016ab8c2b2cab779f45ac2ecd409f3742 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8327: [HUDI-5361] Propagate all hoodie configs from spark sqlconf, but don't overwrite values already set

2023-08-09 Thread via GitHub
hudi-bot commented on PR #8327: URL: https://github.com/apache/hudi/pull/8327#issuecomment-1672232135 ## CI report: * b3388a3bb559227d2415e747681326f6109b4cc2 Azure:

[GitHub] [hudi] neeruks commented on issue #5348: [SUPPORT]org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20220418194506064

2023-08-09 Thread via GitHub
neeruks commented on issue #5348: URL: https://github.com/apache/hudi/issues/5348#issuecomment-1672230698 I am also getting the same error. I am using Glue to read the CSV file and write it into a Hudi table. py4j.protocol.Py4JJavaError: An error occurred while calling o326.save.

[GitHub] [hudi] neeruks commented on issue #2970: [SUPPORT] Failed to upsert for commit time

2023-08-09 Thread via GitHub
neeruks commented on issue #2970: URL: https://github.com/apache/hudi/issues/2970#issuecomment-1672228563 I am also getting the same error. I am using Glue to read the CSV file and write it into a Hudi table. py4j.protocol.Py4JJavaError: An error occurred while calling o326.save.

[GitHub] [hudi] hudi-bot commented on pull request #9410: [HUDI-6673] Fix Incremental Query Syntax - Spark SQL Core Flow Test

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9410: URL: https://github.com/apache/hudi/pull/9410#issuecomment-1672223948 ## CI report: * a3bd3418eccb373f200139996d34b8cc71913a62 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9411: [HUDI-6674] Add rollback info from metadata table in timeline commands

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9411: URL: https://github.com/apache/hudi/pull/9411#issuecomment-1672224229 ## CI report: * 6a8aa88016ab8c2b2cab779f45ac2ecd409f3742 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #8327: [HUDI-5361] Propagate all hoodie configs from spark sqlconf, but don't overwrite values already set

2023-08-09 Thread via GitHub
hudi-bot commented on PR #8327: URL: https://github.com/apache/hudi/pull/8327#issuecomment-1672221796 ## CI report: * b3388a3bb559227d2415e747681326f6109b4cc2 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9410: [HUDI-6673] Fix Incremental Query Syntax - Spark SQL Core Flow Test

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9410: URL: https://github.com/apache/hudi/pull/9410#issuecomment-1672215254 ## CI report: * a3bd3418eccb373f200139996d34b8cc71913a62 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-6674) Add rollback info from metadata table in timeline commands

2023-08-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6674: - Labels: pull-request-available (was: ) > Add rollback info from metadata table in timeline

[GitHub] [hudi] yihua opened a new pull request, #9411: [HUDI-6674] Add rollback info from metadata table in timeline commands

2023-08-09 Thread via GitHub
yihua opened a new pull request, #9411: URL: https://github.com/apache/hudi/pull/9411 ### Change Logs This PR adds the rollback information from the metadata table to the output of the timeline commands in Hudi CLI, given that metadata data table also encounters more rollbacks now.

[jira] [Updated] (HUDI-6674) Add rollback info from metadata table in timeline commands

2023-08-09 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-6674: Fix Version/s: 0.14.0 > Add rollback info from metadata table in timeline commands >

[jira] [Created] (HUDI-6674) Add rollback info from metadata table in timeline commands

2023-08-09 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-6674: --- Summary: Add rollback info from metadata table in timeline commands Key: HUDI-6674 URL: https://issues.apache.org/jira/browse/HUDI-6674 Project: Apache Hudi Issue

[jira] [Assigned] (HUDI-6674) Add rollback info from metadata table in timeline commands

2023-08-09 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-6674: --- Assignee: Ethan Guo > Add rollback info from metadata table in timeline commands >

[jira] [Updated] (HUDI-6673) Spark SQL core flow test incremental query syntax is wrong

2023-08-09 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-6673: -- Status: In Progress (was: Open) > Spark SQL core flow test incremental query syntax is wrong >

[jira] [Updated] (HUDI-6673) Spark SQL core flow test incremental query syntax is wrong

2023-08-09 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-6673: -- Status: Patch Available (was: In Progress) > Spark SQL core flow test incremental query syntax

[GitHub] [hudi] jonvex opened a new pull request, #9410: [HUDI-6673] Fix Incremental Query Syntax - Spark SQL Core Flow Test

2023-08-09 Thread via GitHub
jonvex opened a new pull request, #9410: URL: https://github.com/apache/hudi/pull/9410 ### Change Logs Test runs now ### Impact Testing for release ### Risk level (write none, low medium or high below) none ### Documentation Update N/A

[jira] [Updated] (HUDI-6673) Spark SQL core flow test incremental query syntax is wrong

2023-08-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6673: - Labels: pull-request-available (was: ) > Spark SQL core flow test incremental query syntax is

[jira] [Created] (HUDI-6673) Spark SQL core flow test incremental query syntax is wrong

2023-08-09 Thread Jonathan Vexler (Jira)
Jonathan Vexler created HUDI-6673: - Summary: Spark SQL core flow test incremental query syntax is wrong Key: HUDI-6673 URL: https://issues.apache.org/jira/browse/HUDI-6673 Project: Apache Hudi

[GitHub] [hudi] hudi-bot commented on pull request #9409: [HUDI-6663] New Parquet File Format remove broadcast to fix performance issue for complex file slices

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9409: URL: https://github.com/apache/hudi/pull/9409#issuecomment-1672081146 ## CI report: * d567d80ea610ed8eca248901d310bd40ae4bf8e5 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9408: [HUDI-6671] Support 'alter table add partition' sql

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9408: URL: https://github.com/apache/hudi/pull/9408#issuecomment-1672081089 ## CI report: * 65e9f9828da86e4558b1830493ead64366e69fae Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9409: [HUDI-6663] New Parquet File Format remove broadcast to fix performance issue for complex file slices

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9409: URL: https://github.com/apache/hudi/pull/9409#issuecomment-1672070399 ## CI report: * d567d80ea610ed8eca248901d310bd40ae4bf8e5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #9407: asyncService log prompt incomplete

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9407: URL: https://github.com/apache/hudi/pull/9407#issuecomment-1672058852 ## CI report: * ce0c6dd5877e222dd64ce5ac6434d81168c08727 Azure:

[GitHub] [hudi] emkornfield commented on issue #9355: [SUPPORT] Problem while reading from BQ tables which are synced on Hudi table

2023-08-09 Thread via GitHub
emkornfield commented on issue #9355: URL: https://github.com/apache/hudi/issues/9355#issuecomment-1672050574 This sounds like the likely cause. The solution that uses a view for compatibility with Hudi is inherently flawed. Using the newly contributed [manifest

[jira] [Assigned] (HUDI-6663) Investigate Bootstrap Performance

2023-08-09 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler reassigned HUDI-6663: - Assignee: Jonathan Vexler > Investigate Bootstrap Performance >

[jira] [Updated] (HUDI-6663) Investigate Bootstrap Performance

2023-08-09 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-6663: -- Status: In Progress (was: Open) > Investigate Bootstrap Performance >

[jira] [Updated] (HUDI-6663) Investigate Bootstrap Performance

2023-08-09 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-6663: -- Status: Patch Available (was: In Progress) > Investigate Bootstrap Performance >

[GitHub] [hudi] jonvex opened a new pull request, #9409: [HUDI-6663] New Parquet File Format remove broadcast to fix performance issue for complex file slices

2023-08-09 Thread via GitHub
jonvex opened a new pull request, #9409: URL: https://github.com/apache/hudi/pull/9409 ### Change Logs Remove the broadcast when sending the file slices. ### Impact 1 TB tpcds bootstrap queries 1-14 performance gap between new file format and fast bootstrap went from

[jira] [Updated] (HUDI-6663) Investigate Bootstrap Performance

2023-08-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6663: - Labels: pull-request-available (was: ) > Investigate Bootstrap Performance >

[GitHub] [hudi] hudi-bot commented on pull request #9408: [HUDI-6671] Support 'alter table add partition' sql

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9408: URL: https://github.com/apache/hudi/pull/9408#issuecomment-1671887323 ## CI report: * 65e9f9828da86e4558b1830493ead64366e69fae Azure:

[GitHub] [hudi] the-other-tim-brown commented on issue #9355: [SUPPORT] Problem while reading from BQ tables which are synced on Hudi table

2023-08-09 Thread via GitHub
the-other-tim-brown commented on issue #9355: URL: https://github.com/apache/hudi/issues/9355#issuecomment-1671851547 @ranjanankur I'm taking a look at this and tracking with the JIRA ticket here as well https://issues.apache.org/jira/browse/HUDI-6672 I've reached out to the Google

[jira] [Created] (HUDI-6672) BigQuery Sync updates while queries running cause failures

2023-08-09 Thread Timothy Brown (Jira)
Timothy Brown created HUDI-6672: --- Summary: BigQuery Sync updates while queries running cause failures Key: HUDI-6672 URL: https://issues.apache.org/jira/browse/HUDI-6672 Project: Apache Hudi

[jira] [Assigned] (HUDI-6672) BigQuery Sync updates while queries running cause failures

2023-08-09 Thread Timothy Brown (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Brown reassigned HUDI-6672: --- Assignee: Timothy Brown > BigQuery Sync updates while queries running cause failures >

[GitHub] [hudi] hudi-bot commented on pull request #9408: [HUDI-6671] Support 'alter table add partition' sql

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9408: URL: https://github.com/apache/hudi/pull/9408#issuecomment-1671830253 ## CI report: * 65e9f9828da86e4558b1830493ead64366e69fae UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #9407: asyncService log prompt incomplete

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9407: URL: https://github.com/apache/hudi/pull/9407#issuecomment-1671830173 ## CI report: * ce0c6dd5877e222dd64ce5ac6434d81168c08727 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9407: asyncService log prompt incomplete

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9407: URL: https://github.com/apache/hudi/pull/9407#issuecomment-1671817231 ## CI report: * ce0c6dd5877e222dd64ce5ac6434d81168c08727 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-6671) Support 'alter table add partition' sql

2023-08-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6671: - Labels: pull-request-available (was: ) > Support 'alter table add partition' sql >

[GitHub] [hudi] wecharyu opened a new pull request, #9408: [HUDI-6671] Support 'alter table add partition' sql

2023-08-09 Thread via GitHub
wecharyu opened a new pull request, #9408: URL: https://github.com/apache/hudi/pull/9408 ### Change Logs Hoodie does not support 'add partition' sql now, so we can not get partitions added by 'add partition' command. In this patch, we implement add partition in Hoodie side: 1. add

[jira] [Created] (HUDI-6671) Support 'alter table add partition' sql

2023-08-09 Thread Wechar (Jira)
Wechar created HUDI-6671: Summary: Support 'alter table add partition' sql Key: HUDI-6671 URL: https://issues.apache.org/jira/browse/HUDI-6671 Project: Apache Hudi Issue Type: Bug

[GitHub] [hudi] empcl opened a new pull request, #9407: asyncService log prompt incomplete

2023-08-09 Thread via GitHub
empcl opened a new pull request, #9407: URL: https://github.com/apache/hudi/pull/9407 ### Change Logs asyncService log prompt incomplete ### Impact asyncService log prompt incomplete ### Risk level (write none, low medium or high below) none

[GitHub] [hudi] hudi-bot commented on pull request #9395: [HUDI-6669] HoodieEngineContext should not use parallel stream with parallelism greater than CPU cores

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9395: URL: https://github.com/apache/hudi/pull/9395#issuecomment-1671580470 ## CI report: * f20fe8b171dc78a61639c1eabd7c5e5b4bbac201 Azure:

[GitHub] [hudi] andreacfm commented on issue #9354: [SUPPORT] HoodieDeltaStreamer fails to load org.apache.spark.sql.execution.datasources.Spark33NestedSchemaPruning

2023-08-09 Thread via GitHub
andreacfm commented on issue #9354: URL: https://github.com/apache/hudi/issues/9354#issuecomment-1671565665 @ad1happy2go when trying to compile for spark 3.3 I get this error: ``` [ERROR] COMPILATION ERROR : [INFO] -

[GitHub] [hudi] stream2000 commented on a diff in pull request #9395: [HUDI-6669] HoodieEngineContext should not use parallel stream with parallelism greater than CPU cores

2023-08-09 Thread via GitHub
stream2000 commented on code in PR #9395: URL: https://github.com/apache/hudi/pull/9395#discussion_r1288537503 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java: ## @@ -102,12 +102,12 @@ public RuntimeContext

[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9209: URL: https://github.com/apache/hudi/pull/9209#issuecomment-1671364778 ## CI report: * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN * 803df61d0d04f7e7403d1177325a365e9bbafab5 Azure:

[GitHub] [hudi] SteNicholas commented on a diff in pull request #9395: [HUDI-6669] HoodieEngineContext should not use parallel stream with parallelism greater than CPU cores

2023-08-09 Thread via GitHub
SteNicholas commented on code in PR #9395: URL: https://github.com/apache/hudi/pull/9395#discussion_r1288493882 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java: ## @@ -102,12 +102,12 @@ public RuntimeContext

[GitHub] [hudi] SteNicholas commented on a diff in pull request #9395: [HUDI-6669] HoodieEngineContext should not use parallel stream with parallelism greater than CPU cores

2023-08-09 Thread via GitHub
SteNicholas commented on code in PR #9395: URL: https://github.com/apache/hudi/pull/9395#discussion_r1288493882 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java: ## @@ -102,12 +102,12 @@ public RuntimeContext

[GitHub] [hudi] hudi-bot commented on pull request #9395: [HUDI-6669] HoodieEngineContext should not use parallel stream with parallelism greater than CPU cores

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9395: URL: https://github.com/apache/hudi/pull/9395#issuecomment-1671265476 ## CI report: * a60f7f89b5377119bf8bef6c7ddfd0dc821de1fc Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9395: [HUDI-6669] HoodieEngineContext should not use parallel stream with parallelism greater than CPU cores

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9395: URL: https://github.com/apache/hudi/pull/9395#issuecomment-1671200936 ## CI report: * a60f7f89b5377119bf8bef6c7ddfd0dc821de1fc Azure:

[GitHub] [hudi] stream2000 commented on a diff in pull request #9395: [HUDI-6669] HoodieEngineContext should not use parallel stream with parallelism greater than CPU cores

2023-08-09 Thread via GitHub
stream2000 commented on code in PR #9395: URL: https://github.com/apache/hudi/pull/9395#discussion_r1288371818 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java: ## @@ -102,12 +102,12 @@ public RuntimeContext

[GitHub] [hudi] hudi-bot commented on pull request #9403: Added kafka key as part of hudi metadata columns for Json & Avro KafkaSource

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9403: URL: https://github.com/apache/hudi/pull/9403#issuecomment-1671187660 ## CI report: * 55da0942b542c664e49c7ab9ca9698dfbf67968e Azure:

[GitHub] [hudi] SteNicholas commented on pull request #9395: [HUDI-6669] HoodieEngineContext should not use parallel stream with parallelism greater than CPU cores

2023-08-09 Thread via GitHub
SteNicholas commented on PR #9395: URL: https://github.com/apache/hudi/pull/9395#issuecomment-1671137882 @stream2000, thanks for the fix. I have rebased the lastest master branch. cc @danny0405. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] SteNicholas commented on a diff in pull request #9395: [HUDI-6669] HoodieEngineContext should not use parallel stream with parallelism greater than CPU cores

2023-08-09 Thread via GitHub
SteNicholas commented on code in PR #9395: URL: https://github.com/apache/hudi/pull/9395#discussion_r1288326850 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java: ## @@ -102,12 +102,12 @@ public RuntimeContext

[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9209: URL: https://github.com/apache/hudi/pull/9209#issuecomment-1671129502 ## CI report: * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN * 57c1b843608a9b63d143ead5dd5168613bb13969 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9209: URL: https://github.com/apache/hudi/pull/9209#issuecomment-1671117746 ## CI report: * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN * 57c1b843608a9b63d143ead5dd5168613bb13969 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #9405: [HUDI-6670] Fix timeline check in metadata table validator

2023-08-09 Thread via GitHub
hudi-bot commented on PR #9405: URL: https://github.com/apache/hudi/pull/9405#issuecomment-1671103541 ## CI report: * fc027c28476d50737566c3b714a4d58c38c39ff9 Azure:

[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-08-09 Thread via GitHub
danny0405 commented on code in PR #9209: URL: https://github.com/apache/hudi/pull/9209#discussion_r1288283343 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java: ## @@ -18,75 +18,127 @@ package

[GitHub] [hudi] stream2000 commented on a diff in pull request #9395: [HUDI-6669] HoodieEngineContext should not use parallel stream with parallelism greater than CPU cores

2023-08-09 Thread via GitHub
stream2000 commented on code in PR #9395: URL: https://github.com/apache/hudi/pull/9395#discussion_r1288224103 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/common/HoodieFlinkEngineContext.java: ## @@ -102,12 +102,12 @@ public RuntimeContext

[GitHub] [hudi] stream2000 commented on pull request #9395: [HUDI-6669] HoodieEngineContext should not use parallel stream with parallelism greater than CPU cores

2023-08-09 Thread via GitHub
stream2000 commented on PR #9395: URL: https://github.com/apache/hudi/pull/9395#issuecomment-1670984818 @SteNicholas Hi, sorry for the failure ci introduced. Now we can rebase the lastest master and test -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] leesf merged pull request #9401: [MINOR] Fix consistent hashing bucket index it failure

2023-08-09 Thread via GitHub
leesf merged PR #9401: URL: https://github.com/apache/hudi/pull/9401 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[hudi] branch master updated: [MINOR] Fix consistent hashing bucket index FT failure (#9401)

2023-08-09 Thread leesf
This is an automated email from the ASF dual-hosted git repository. leesf pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 9b22583dbe0 [MINOR] Fix consistent hashing bucket

[GitHub] [hudi] leesf commented on pull request #9401: [MINOR] Fix consistent hashing bucket index it failure

2023-08-09 Thread via GitHub
leesf commented on PR #9401: URL: https://github.com/apache/hudi/pull/9401#issuecomment-1670980192 +1 as the FT spark-client passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-08-09 Thread via GitHub
danny0405 commented on code in PR #9209: URL: https://github.com/apache/hudi/pull/9209#discussion_r1288178605 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java: ## @@ -0,0 +1,382 @@ +/* + * Licensed to the Apache Software

[jira] [Comment Edited] (HUDI-3425) Clean up spill path created by Hudi during uneventful shutdown

2023-08-09 Thread Xinglong Wang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752332#comment-17752332 ] Xinglong Wang edited comment on HUDI-3425 at 8/9/23 8:57 AM: - {{I have

[GitHub] [hudi] aib628 commented on issue #8848: [SUPPORT] Hive Sync tool fails to sync Hoodi table written using Flink 1.16 to HMS

2023-08-09 Thread via GitHub
aib628 commented on issue #8848: URL: https://github.com/apache/hudi/issues/8848#issuecomment-1670938476 @danny0405 Yeah, i'm using hadoop3.1.0 + hive 3.1.2 package it from source, and deploy it using docker image of 'apachehudi/hudi-hadoop_3.1.0-hive_3.1.2:latest'. -- This is an

[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-08-09 Thread via GitHub
danny0405 commented on code in PR #9209: URL: https://github.com/apache/hudi/pull/9209#discussion_r1288171581 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java: ## @@ -0,0 +1,382 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-08-09 Thread via GitHub
danny0405 commented on code in PR #9209: URL: https://github.com/apache/hudi/pull/9209#discussion_r1288170683 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java: ## @@ -0,0 +1,382 @@ +/* + * Licensed to the Apache Software

[jira] [Commented] (HUDI-3425) Clean up spill path created by Hudi during uneventful shutdown

2023-08-09 Thread Xinglong Wang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752332#comment-17752332 ] Xinglong Wang commented on HUDI-3425: - I have encountered the same problem. I am using Flink on Yarn.

[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-08-09 Thread via GitHub
danny0405 commented on code in PR #9209: URL: https://github.com/apache/hudi/pull/9209#discussion_r1288167338 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java: ## @@ -0,0 +1,382 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-08-09 Thread via GitHub
danny0405 commented on code in PR #9209: URL: https://github.com/apache/hudi/pull/9209#discussion_r1288155900 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java: ## @@ -0,0 +1,382 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

2023-08-09 Thread via GitHub
danny0405 commented on code in PR #9209: URL: https://github.com/apache/hudi/pull/9209#discussion_r1288155900 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java: ## @@ -0,0 +1,382 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] bhasudha commented on pull request #9406: [DOCS] Update Metadata table and metadata indexing related pages

2023-08-09 Thread via GitHub
bhasudha commented on PR #9406: URL: https://github.com/apache/hudi/pull/9406#issuecomment-1670918525 @codope Just FYI. This PR can be reviewed but must be merged after this [PR](https://github.com/apache/hudi/pull/9372) is merged for dependency on page links. -- This is an automated

  1   2   >