[GitHub] [hudi] xushiyan commented on issue #7431: Metastore connection is closed properly

2022-12-13 Thread GitBox
xushiyan commented on issue #7431: URL: https://github.com/apache/hudi/issues/7431#issuecomment-1348212855 @njalan interesting. when you say "suddenly faced that the many spark jobs got stuck after Hive sync completed" meaning you did not change anything or have any deployments of the

[GitHub] [hudi] xushiyan commented on issue #7406: [SUPPORT] Support Debezium JSON

2022-12-13 Thread GitBox
xushiyan commented on issue #7406: URL: https://github.com/apache/hudi/issues/7406#issuecomment-1348512844 @melin can you elaborate the use case pls? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[hudi] branch master updated: [HUDI-5366] Closing metadata writer from within writeClient (#7437)

2022-12-13 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new e2c7f78d940 [HUDI-5366] Closing metadata writer

[GitHub] [hudi] codope merged pull request #7437: [HUDI-5366] Closing metadata writer from within writeClient

2022-12-13 Thread GitBox
codope merged PR #7437: URL: https://github.com/apache/hudi/pull/7437 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] hudi-bot commented on pull request #7366: [HUDI-5318] Fix partition pruning for clustering scheduling

2022-12-13 Thread GitBox
hudi-bot commented on PR #7366: URL: https://github.com/apache/hudi/pull/7366#issuecomment-1348996735 ## CI report: * 3f6572349834d904a697fbd8c8546f56a7f2844a Azure:

[jira] [Updated] (HUDI-5262) When creating table in spark-sql setting wrong keygenerator config does not warn

2022-12-13 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-5262: -- Sprint: 2022/12/06 > When creating table in spark-sql setting wrong keygenerator config does

[GitHub] [hudi] sstimmel commented on issue #7409: [SUPPORT]

2022-12-13 Thread GitBox
sstimmel commented on issue #7409: URL: https://github.com/apache/hudi/issues/7409#issuecomment-1348896935 > @sstimmel this is some dependency conflicts, likely caused by hudi-cli or hudi-hive-sync-bundle. can you try removing these 2 and only leaving spark-bundle and utilities-slim

[jira] [Updated] (HUDI-5376) Update quickstart guide for hudi hoodie.datasource.write.keygenerator.class spark-sql change

2022-12-13 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-5376: -- Sprint: 2022/12/06 > Update quickstart guide for hudi

[jira] [Updated] (HUDI-5376) Update quickstart guide for hudi hoodie.datasource.write.keygenerator.class spark-sql change

2022-12-13 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-5376: -- Status: Patch Available (was: In Progress) > Update quickstart guide for hudi

[jira] [Updated] (HUDI-5376) Update quickstart guide for hudi hoodie.datasource.write.keygenerator.class spark-sql change

2022-12-13 Thread Jonathan Vexler (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-5376: -- Status: In Progress (was: Open) > Update quickstart guide for hudi

[hudi] branch master updated: [HUDI-4432] Checkpoint management for muti-writer scenario (#7383)

2022-12-13 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 2b6688944e7 [HUDI-4432] Checkpoint management for

[GitHub] [hudi] codope merged pull request #7383: [HUDI-4432] Checkpoint management for muti-writer scenario

2022-12-13 Thread GitBox
codope merged PR #7383: URL: https://github.com/apache/hudi/pull/7383 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Updated] (HUDI-5383) Test 0.12.2 release branch

2022-12-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5383: - Labels: pull-request-available (was: ) > Test 0.12.2 release branch >

[GitHub] [hudi] hudi-bot commented on pull request #7394: [HUDI-5262] Allow hoodie.datasource.write.keygenerator.class to be used in spark-sql create table

2022-12-13 Thread GitBox
hudi-bot commented on PR #7394: URL: https://github.com/apache/hudi/pull/7394#issuecomment-1349012854 ## CI report: * 43a31c8ce9849f487e521c1c9b467dd4eada6331 UNKNOWN * 6d3d125caa257a3b290ae286dd77499a39683750 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7423: [MINOR] Adding optimization rule to appropriately push down filters into the `HoodieFileIndex`

2022-12-13 Thread GitBox
hudi-bot commented on PR #7423: URL: https://github.com/apache/hudi/pull/7423#issuecomment-1349318087 ## CI report: * 4c28887b7079a7e00ca0543a7ac3daee9872422b Azure:

[GitHub] [hudi] jonvex commented on issue #7294: [SUPPORT] Different keygen class assigned by Hudi in 0.11.1 and 0.12.1 while creating a table with multiple primary keys

2022-12-13 Thread GitBox
jonvex commented on issue #7294: URL: https://github.com/apache/hudi/issues/7294#issuecomment-1349465490 I just tried out my test with options instead of tblproperties and it still passed. So not sure what else there is to try -- This is an automated message from the Apache Git Service.

[hudi] 04/04: Fixing test failure

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit 4d0ea23ee64978d188557f5f8546d436c9083500 Author: sivabalan AuthorDate: Tue Dec 13 10:35:56 2022 -0800

[hudi] 01/04: [MINOR] Disable the `SparkSqlCoreFlow` tests (#7368)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit 6912c5a951122b3681de5b7733d2ba0c3caeb50b Author: Jon Vexler AuthorDate: Fri Dec 2 22:16:46 2022 -0500

[hudi] 03/04: [HUDI-5331] Add schema settings with stream api (#7384)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit b62a4c9aac0c742106187139497ad32433f84dc2 Author: superche

[hudi] 02/04: [HUDI-5179] Updated Hudi Release guide (#7212)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit 78bbb8a61a62d558593c762baa7402c9e619019c Author: Zhaojing Yu AuthorDate: Tue Dec 6 08:04:54 2022 +0800

[hudi] branch release-0.12.2-shadow updated (89eb565a939 -> 4d0ea23ee64)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git from 89eb565a939 [HUDI-5302] Fix: compute hash key from recordKey failed when recordKeyValue contains ','

[GitHub] [hudi] sstimmel commented on issue #7409: [SUPPORT]

2022-12-13 Thread GitBox
sstimmel commented on issue #7409: URL: https://github.com/apache/hudi/issues/7409#issuecomment-1349025430 i removed hudi-sync-bundle and switched over to use hudi-utilities-bundle instead of hudi-utilities-slim-bundle, since that has hive-sync included, but still see the error with that

[jira] [Created] (HUDI-5383) Test 0.12.2 release branch

2022-12-13 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-5383: - Summary: Test 0.12.2 release branch Key: HUDI-5383 URL: https://issues.apache.org/jira/browse/HUDI-5383 Project: Apache Hudi Issue Type: Test

[GitHub] [hudi] soumilshah1995 commented on issue #7430: [BUG] MOR Table Hard Deletes Create issue with Athena Querying RT Tables

2022-12-13 Thread GitBox
soumilshah1995 commented on issue #7430: URL: https://github.com/apache/hudi/issues/7430#issuecomment-1349553479 Hi Version of Glue used is 4.0 glue 4.0 natively support HUDI i am not aware behind the scene which version Glue uses for Apache HUDI Here are steps in PDF

[hudi] 05/09: [HUDI-5304] Disabling spark-sql core flow tests to unblock CI (#7346)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit d54ecdfcab09445cafca022ba424cbdffafbf1d4 Author: Sivabalan Narayanan AuthorDate: Wed Nov 30 18:20:22

[hudi] 04/09: [MINOR] Bumping Azure Ubuntu image to 22.04, as 18.04 will be deprecated soon (#7347)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit 9ae6692a3db2c94d929271362cfd473703fd2d8f Author: Alexey Kudinkin AuthorDate: Wed Nov 30 13:35:43 2022

[hudi] 02/09: Rebased MOR iterators onto a `CachingIterator` (to be idempotent) (#7334)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit eb8566ed654b483fdefa54dfe0a1f79cf4f4204f Author: Alexey Kudinkin AuthorDate: Wed Nov 30 13:31:33 2022

[hudi] 06/09: [HUDI-5306] Unify RecordIterator and HoodieParquetReader with ClosableIterator (#7340)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit 1b8e5ec12e300ed76c74fdddb585ede55c66b28a Author: Danny Chan AuthorDate: Thu Dec 1 17:13:59 2022 +0800

[hudi] 03/09: resolving conflicts for #7334

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit f796f02791fb792a94ef7b652f2ee862d454a5b7 Author: sivabalan AuthorDate: Tue Dec 13 07:28:26 2022 -0800

[hudi] branch release-0.12.2-shadow created (now 89eb565a939)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git at 89eb565a939 [HUDI-5302] Fix: compute hash key from recordKey failed when recordKeyValue contains ','

[hudi] 07/09: resolving conflicts for HUDI-5306

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit c30e60fe93953dd279a9ef27a40174b4728b937c Author: sivabalan AuthorDate: Tue Dec 13 08:26:19 2022 -0800

[hudi] 08/09: Revert "[MINOR] Bumping Azure Ubuntu image to 22.04, as 18.04 will be deprecated soon (#7347)" (#7350)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit 838887f577c717a381d75a331f96dfa2ec7fef54 Author: Alexey Kudinkin AuthorDate: Thu Dec 1 02:03:39 2022

[hudi] 09/09: [HUDI-5302] Fix: compute hash key from recordKey failed when recordKeyValue contains ',' (#7342)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit 89eb565a939ef91872fe4ff1a803dd03e8154888 Author: shaoxiong.zhan

[hudi] 01/09: [HUDI-5279] move logic for deleting active instant to HoodieActiveTimeline (#7196)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit a0ab4014936166d388e73e151d35de6205a97f9e Author: Yann Byron AuthorDate: Wed Nov 30 18:11:23 2022 +0800

[GitHub] [hudi] hudi-bot commented on pull request #7394: [HUDI-5262] Allow hoodie.datasource.write.keygenerator.class to be used in spark-sql create table

2022-12-13 Thread GitBox
hudi-bot commented on PR #7394: URL: https://github.com/apache/hudi/pull/7394#issuecomment-1349317712 ## CI report: * 43a31c8ce9849f487e521c1c9b467dd4eada6331 UNKNOWN * 6d3d125caa257a3b290ae286dd77499a39683750 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7413: [HUDI-5321] correctly implement arePartitionRecordsSorted for bulk insert ColumnSortPartitioners

2022-12-13 Thread GitBox
hudi-bot commented on PR #7413: URL: https://github.com/apache/hudi/pull/7413#issuecomment-1349317944 ## CI report: * 52dd21d8ee01c77a92f596e9e2205d9e25fc72eb Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7345: [HUDI-3378] RFC46 rebase

2022-12-13 Thread GitBox
hudi-bot commented on PR #7345: URL: https://github.com/apache/hudi/pull/7345#issuecomment-1349317274 ## CI report: * e5f1dba84479f08417f25f53a79f6dae4425ba23 UNKNOWN * 2cfcca5c4f1a3a17b68d50f605f736c3a03c2e3f UNKNOWN * 1930cfe77fc3ddbd75564a75558b1211f823be89 Azure:

[GitHub] [hudi] nsivabalan opened a new pull request, #7447: [HUDI-5383] Testing 0.12.2 release branch for CI run

2022-12-13 Thread GitBox
nsivabalan opened a new pull request, #7447: URL: https://github.com/apache/hudi/pull/7447 ### Change Logs Testing 0.12.2 release branch ### Impact n/a ### Risk level (write none, low medium or high below) low ### Documentation Update N/A

[hudi] 05/10: [HUDI-4764] AWS GlueSync turn partition already exist error into warning (#6505)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit ed989b6ef4317553ff891f21ff6c7be636cf7772 Author: Nicolas Paris AuthorDate: Wed Dec 7 14:00:56 2022

[hudi] 07/10: [HUDI-3661] Flink async compaction is not thread safe when use watermark (#7399)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit 49e6976bc48c60c5bf8c50d8708d18eaec32deb5 Author: Danny Chan AuthorDate: Wed Dec 7 18:31:26 2022 +0800

[hudi] 10/10: [HUDI-5295] One meta sync failure should not prevent other meta sync from occurring (#7367)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit 12a0897eac9ca7059da44c9cb6e625030493087c Author: Jon Vexler AuthorDate: Wed Dec 7 22:23:34 2022 -0500

[hudi] 06/10: [HUDI-5163] Fix failure handling with spark datasource write (#7140)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit 245574326ccb5a8d0f21667ae1776b200254e41a Author: Sivabalan Narayanan AuthorDate: Wed Dec 7 09:17:27

[hudi] branch release-0.12.2-shadow updated (4d0ea23ee64 -> 12a0897eac9)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git from 4d0ea23ee64 Fixing test failure new 8cf1b3f1530 [MINOR] Fix locale specific

[hudi] 01/10: [MINOR] Fix locale specific NumberFormatException in testutils HoodieTestDataGenerator (#7215)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit 8cf1b3f15304189aa3b356ef615c257e177842ed Author: Alexander Trushev AuthorDate: Wed Dec 7 13:42:14 2022

[hudi] 02/10: [HUDI-5334] Fix checkpoint reading for structured streaming (#7389)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit 1b4b40e44a176cefcb692c85a2242cc539348c17 Author: Shiyan Xu <2701446+xushi...@users.noreply.github.com>

[hudi] 04/10: [HUDI-5314] add call help procedure (#7361)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit 89370ae36846eabd2623284426e0a2df4bc746bc Author: 苏承祥 AuthorDate: Wed Dec 7 20:03:30 2022 +0800

[hudi] 03/10: [HUDI-5290] Remove the lock in HoodieFlinkWriteClient#writeTableMetadata (#7320)

2022-12-13 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch release-0.12.2-shadow in repository https://gitbox.apache.org/repos/asf/hudi.git commit 135fc9e1bcb1c9ab5e50e12dd6c75bd4aac19985 Author: just-JL AuthorDate: Wed Dec 7 19:45:49 2022 +0800

[GitHub] [hudi] hudi-bot commented on pull request #7174: [HUDI-5023] Consuming records from Iterator directly instead of using inner message queue

2022-12-13 Thread GitBox
hudi-bot commented on PR #7174: URL: https://github.com/apache/hudi/pull/7174#issuecomment-1348190491 ## CI report: * 0cd32cd2d101d3b9ec0fdeb4463942ad0f00a50b UNKNOWN * db06b33b3f02d25e034d7df8d129f6ab6b643bb2 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7159: [HUDI-5173]Skip if there is only one file in clusteringGroup

2022-12-13 Thread GitBox
hudi-bot commented on PR #7159: URL: https://github.com/apache/hudi/pull/7159#issuecomment-1348190390 ## CI report: * 15ecd91180d32c7fa1905c11408f4bc23347e682 UNKNOWN * 80e5351530c18e0fa21c3fab29934e980a2f1317 Azure:

[GitHub] [hudi] xushiyan commented on issue #7422: [SUPPORT] Write location changes are not propagated by hive sync

2022-12-13 Thread GitBox
xushiyan commented on issue #7422: URL: https://github.com/apache/hudi/issues/7422#issuecomment-1348269788 @blrnw3 have you tried upgrade to later version of EMR which contains later Hudi? -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hudi] xushiyan commented on issue #7409: [SUPPORT]

2022-12-13 Thread GitBox
xushiyan commented on issue #7409: URL: https://github.com/apache/hudi/issues/7409#issuecomment-1348497185 @sstimmel this is some dependency conflicts, likely caused by hudi-cli or hudi-hive-sync-bundle. can you try removing these 2 and only leaving spark-bundle and utilities-slim bundle?

[GitHub] [hudi] hudi-bot commented on pull request #7440: [HUDI-5377] Add call stack information to lock file

2022-12-13 Thread GitBox
hudi-bot commented on PR #7440: URL: https://github.com/apache/hudi/pull/7440#issuecomment-1348585160 ## CI report: * 67e64ca0d35342d303f5c0027db72ec4c14f1890 UNKNOWN * 3a660182f2351faa568087baa1de216d4151702a Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7445: [HUDI-5380] Fixing change table path but table location in metastore …

2022-12-13 Thread GitBox
hudi-bot commented on PR #7445: URL: https://github.com/apache/hudi/pull/7445#issuecomment-1348610051 ## CI report: * 357325b2ea7227e273d713a1f6d71ebc08e6ce0f Azure:

[jira] [Created] (HUDI-5382) hoodie.datasource.write.partitionpath.field is inconsistent in the document

2022-12-13 Thread Akira Ajisaka (Jira)
Akira Ajisaka created HUDI-5382: --- Summary: hoodie.datasource.write.partitionpath.field is inconsistent in the document Key: HUDI-5382 URL: https://issues.apache.org/jira/browse/HUDI-5382 Project:

[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #7440: [HUDI-5377] Add call stack information to lock file

2022-12-13 Thread GitBox
XuQianJin-Stars commented on code in PR #7440: URL: https://github.com/apache/hudi/pull/7440#discussion_r1047026737 ## hudi-common/src/main/java/org/apache/hudi/common/lock/LockProvider.java: ## @@ -50,6 +50,10 @@ default T getLock() { throw new IllegalArgumentException();

[GitHub] [hudi] voonhous opened a new issue, #7444: [SUPPORT] Implicit schema changes supported by Avro schema-resolution will

2022-12-13 Thread GitBox
voonhous opened a new issue, #7444: URL: https://github.com/apache/hudi/issues/7444 - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? (Yes) - Join the mailing list to engage in conversations and get faster support at dev-subscr...@hudi.apache.org. - If

[GitHub] [hudi] wzx140 commented on pull request #7345: [HUDI-3378] RFC46 rebase

2022-12-13 Thread GitBox
wzx140 commented on PR #7345: URL: https://github.com/apache/hudi/pull/7345#issuecomment-1348517823 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] xushiyan commented on issue #7392: [SUPPORT] Unable to read data from MOR table using spark. ERROR: org.apache.spark.sql.execution.datasources.PartitionedFile

2022-12-13 Thread GitBox
xushiyan commented on issue #7392: URL: https://github.com/apache/hudi/issues/7392#issuecomment-1348571338 this is due to amzn spark is different from open source spark in some APIs like PartitionedFile. when you use Glue, it's an aws-managed service. as you're using hudi 0.10.0 which is

[GitHub] [hudi] zhangyue19921010 commented on pull request #7174: [HUDI-5023] Consuming records from Iterator directly instead of using inner message queue

2022-12-13 Thread GitBox
zhangyue19921010 commented on PR #7174: URL: https://github.com/apache/hudi/pull/7174#issuecomment-1348569594 Hey @alexeykudinkin All comments addressed. PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] codope commented on issue #7133: lazyReading affect

2022-12-13 Thread GitBox
codope commented on issue #7133: URL: https://github.com/apache/hudi/issues/7133#issuecomment-1348732865 @china-shang please close the issue if your query is answered. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[hudi] branch master updated: [HUDI-4113] Fix cannot parse schema when use spark delete sql (#5610)

2022-12-13 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 5faa36f99ca [HUDI-4113] Fix cannot parse schema

[GitHub] [hudi] codope merged pull request #5610: [HUDI-4113] Fix cannot parse schema when use spark delete sql

2022-12-13 Thread GitBox
codope merged PR #5610: URL: https://github.com/apache/hudi/pull/5610 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] codope commented on issue #6014: [SUPPORT] High runtime for a batch in SparkWriteHelper stage

2022-12-13 Thread GitBox
codope commented on issue #6014: URL: https://github.com/apache/hudi/issues/6014#issuecomment-1348769702 @veenaypatil Did the suggestion above work for you? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] codope closed issue #5492: _hoodie_is_delete works differently on hudi spark datasource on docker compare to hudi on emr.

2022-12-13 Thread GitBox
codope closed issue #5492: _hoodie_is_delete works differently on hudi spark datasource on docker compare to hudi on emr. URL: https://github.com/apache/hudi/issues/5492 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [hudi] danny0405 commented on a diff in pull request #7159: [HUDI-5173]Skip if there is only one file in clusteringGroup

2022-12-13 Thread GitBox
danny0405 commented on code in PR #7159: URL: https://github.com/apache/hudi/pull/7159#discussion_r1046888393 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/cdc/TestCDCDataFrameSuite.scala: ## @@ -118,6 +118,7 @@ class TestCDCDataFrameSuite

[GitHub] [hudi] zhuanshenbsj1 commented on a diff in pull request #7159: [HUDI-5173]Skip if there is only one file in clusteringGroup

2022-12-13 Thread GitBox
zhuanshenbsj1 commented on code in PR #7159: URL: https://github.com/apache/hudi/pull/7159#discussion_r1046889456 ## hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java: ## @@ -141,6 +141,7 @@ private void setHiveColumnNameProps(List

[GitHub] [hudi] codope commented on issue #6869: [SUPPORT] Incremental upsert or merge is not working

2022-12-13 Thread GitBox
codope commented on issue #6869: URL: https://github.com/apache/hudi/issues/6869#issuecomment-1348766492 @gtwuser any updates for us? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] codope commented on issue #5492: _hoodie_is_delete works differently on hudi spark datasource on docker compare to hudi on emr.

2022-12-13 Thread GitBox
codope commented on issue #5492: URL: https://github.com/apache/hudi/issues/5492#issuecomment-1348767380 Closing due to inactivity -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] stream2000 commented on pull request #7366: [HUDI-5318] Fix partition pruning for clustering scheduling

2022-12-13 Thread GitBox
stream2000 commented on PR #7366: URL: https://github.com/apache/hudi/pull/7366#issuecomment-1348080989 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] njalan commented on issue #7431: Metastore connection is closed properly

2022-12-13 Thread GitBox
njalan commented on issue #7431: URL: https://github.com/apache/hudi/issues/7431#issuecomment-1348473111 It just happened once with one and half years. I also disabled that hive meta sync for spark streaming job since each micro will do hive meta sync but no schema changed. By now it is

[GitHub] [hudi] xushiyan commented on issue #7414: [SUPPORT] Support lsm tree writing

2022-12-13 Thread GitBox
xushiyan commented on issue #7414: URL: https://github.com/apache/hudi/issues/7414#issuecomment-1348484592 @waywtdcc would you like to draft an RFC on this topic to illustrate more details? -- This is an automated message from the Apache Git Service. To respond to the message, please log

[jira] [Updated] (HUDI-5380) Change table path but table location in metastore will not change after hive-sync.

2022-12-13 Thread Ying Lin (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ying Lin updated HUDI-5380: --- Summary: Change table path but table location in metastore will not change after hive-sync. (was: If we

[GitHub] [hudi] BruceKellan commented on issue #7422: [SUPPORT] Write location changes are not propagated by hive sync

2022-12-13 Thread GitBox
BruceKellan commented on issue #7422: URL: https://github.com/apache/hudi/issues/7422#issuecomment-1348503578 This problem seems to still exist in the master branch, I will open a PR to fix it. -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hudi] njalan commented on issue #7431: Metastore connection is closed properly

2022-12-13 Thread GitBox
njalan commented on issue #7431: URL: https://github.com/apache/hudi/issues/7431#issuecomment-1348502582 I have like around 500 jobs with two metastore servers on two vms, one vm is like 16G for hive metastore. Do you think these two metastore servier is enough? -- This is an automated

[GitHub] [hudi] BruceKellan commented on pull request #7445: [HUDI-5380] Fixing change table path but table location in metastore …

2022-12-13 Thread GitBox
BruceKellan commented on PR #7445: URL: https://github.com/apache/hudi/pull/7445#issuecomment-1348504228 Related: #7422 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[jira] [Created] (HUDI-5381) Class cast exception with Flink 1.15 source when reading table written using bulk insert

2022-12-13 Thread Kenneth William Krugler (Jira)
Kenneth William Krugler created HUDI-5381: - Summary: Class cast exception with Flink 1.15 source when reading table written using bulk insert Key: HUDI-5381 URL:

[GitHub] [hudi] codope commented on issue #7446: [SUPPORT] is it possible to read/write hudi files with another programming language?

2022-12-13 Thread GitBox
codope commented on issue #7446: URL: https://github.com/apache/hudi/issues/7446#issuecomment-1348614349 Not yet, but it's planned for version 1.0.0. https://hudi.apache.org/roadmap/ Currently, one can use Hudi with Python (pyspark), Java and Scala. -- This is an automated

[GitHub] [hudi] codope commented on issue #7081: [SUPPORT] optimistic_concurrency_control

2022-12-13 Thread GitBox
codope commented on issue #7081: URL: https://github.com/apache/hudi/issues/7081#issuecomment-1348737794 Closing due to inactivity. Please reopen if you're still facing the issue after suggested config. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] codope closed issue #7081: [SUPPORT] optimistic_concurrency_control

2022-12-13 Thread GitBox
codope closed issue #7081: [SUPPORT] optimistic_concurrency_control URL: https://github.com/apache/hudi/issues/7081 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] danny0405 commented on a diff in pull request #7159: [HUDI-5173]Skip if there is only one file in clusteringGroup

2022-12-13 Thread GitBox
danny0405 commented on code in PR #7159: URL: https://github.com/apache/hudi/pull/7159#discussion_r1046886068 ## hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/realtime/TestHoodieRealtimeRecordReader.java: ## @@ -141,6 +141,7 @@ private void setHiveColumnNameProps(List

[GitHub] [hudi] hudi-bot commented on pull request #7174: [HUDI-5023] Consuming records from Iterator directly instead of using inner message queue

2022-12-13 Thread GitBox
hudi-bot commented on PR #7174: URL: https://github.com/apache/hudi/pull/7174#issuecomment-1348075975 ## CI report: * 0cd32cd2d101d3b9ec0fdeb4463942ad0f00a50b UNKNOWN * db06b33b3f02d25e034d7df8d129f6ab6b643bb2 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7437: [HUDI-5366] Closing metadata writer from within writeClient

2022-12-13 Thread GitBox
hudi-bot commented on PR #7437: URL: https://github.com/apache/hudi/pull/7437#issuecomment-1348183315 ## CI report: * 40c69ac7d433245f25296fd2883205c890596dd9 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7366: [HUDI-5318] Fix partition pruning for clustering scheduling

2022-12-13 Thread GitBox
hudi-bot commented on PR #7366: URL: https://github.com/apache/hudi/pull/7366#issuecomment-1348182943 ## CI report: * 3f6572349834d904a697fbd8c8546f56a7f2844a Azure:

[GitHub] [hudi] xushiyan commented on issue #7430: [BUG] MOR Table Hard Deletes Create issue with Athena Querying RT Tables

2022-12-13 Thread GitBox
xushiyan commented on issue #7430: URL: https://github.com/apache/hudi/issues/7430#issuecomment-1348230816 @lokeshj1703 let's try to reproduce this. @soumilshah1995 is this observed on master version? or 0.12.1? -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] BruceKellan opened a new pull request, #7445: [HUDI-5380] Fixing change table path but table location in metastore …

2022-12-13 Thread GitBox
BruceKellan opened a new pull request, #7445: URL: https://github.com/apache/hudi/pull/7445 …will not change after hive-sync. ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or

[jira] [Updated] (HUDI-5380) Change table path but table location in metastore will not change after hive-sync.

2022-12-13 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5380: - Labels: pull-request-available (was: ) > Change table path but table location in metastore will

[GitHub] [hudi] schlichtanders opened a new issue, #7446: [SUPPORT] is it possible to read/write hudi files with another programming language?

2022-12-13 Thread GitBox
schlichtanders opened a new issue, #7446: URL: https://github.com/apache/hudi/issues/7446 Hi, I am curious about the state of hudi. We are currently using it via Spark, however thinking about switching to another language. Is it possible to write Hudi files via C, C++, Rust,

[GitHub] [hudi] codope commented on issue #7444: [SUPPORT] Implicit schema changes supported by Avro schema-resolution will not work properly if there are filegroups with old schema

2022-12-13 Thread GitBox
codope commented on issue #7444: URL: https://github.com/apache/hudi/issues/7444#issuecomment-1348605028 @voonhous Thanks for sharing a test to reproduce the issue! @xiarixiaoyao Can you please take a look? -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] codope commented on issue #6503: [SUPPORT] Hudi Merge Into with larger volume

2022-12-13 Thread GitBox
codope commented on issue #6503: URL: https://github.com/apache/hudi/issues/6503#issuecomment-1348712592 Should be fixed in master via https://github.com/apache/hudi/commit/5b9fcc4540b953a3f4af22f1cb2ed28e88849069 -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] codope closed issue #6503: [SUPPORT] Hudi Merge Into with larger volume

2022-12-13 Thread GitBox
codope closed issue #6503: [SUPPORT] Hudi Merge Into with larger volume URL: https://github.com/apache/hudi/issues/6503 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] codope commented on issue #7221: [SUPPORT] Spark could not read Flink created table

2022-12-13 Thread GitBox
codope commented on issue #7221: URL: https://github.com/apache/hudi/issues/7221#issuecomment-1348723532 @punish-yh gentle ping to check the recommended solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hudi] codope commented on issue #5673: [SUPPORT]scala.MatchError for rename colum

2022-12-13 Thread GitBox
codope commented on issue #5673: URL: https://github.com/apache/hudi/issues/5673#issuecomment-1348758030 @sunke38 Gentle reminder to try out the suggestion and close if it works. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[hudi] 02/17: Revert "[MINOR] Bumping Azure Ubuntu image to 22.04, as 18.04 will be deprecated soon (#7347)" (#7350)

2022-12-13 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch release-0.12.2 in repository https://gitbox.apache.org/repos/asf/hudi.git commit 12ef312730935fc9823a781d5d3763029a2dd4c3 Author: Alexey Kudinkin AuthorDate: Thu Dec 1 02:03:39 2022 -0800

[hudi] 17/17: [HUDI-5295] One meta sync failure should not prevent other meta sync from occurring (#7367)

2022-12-13 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch release-0.12.2 in repository https://gitbox.apache.org/repos/asf/hudi.git commit b9109cd9fad02910194fc8e10e78f5cd1fc4aeb7 Author: Jon Vexler AuthorDate: Wed Dec 7 22:23:34 2022 -0500

[hudi] 01/17: [HUDI-5306] Unify RecordIterator and HoodieParquetReader with ClosableIterator (#7340)

2022-12-13 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch release-0.12.2 in repository https://gitbox.apache.org/repos/asf/hudi.git commit d228a10770ac97f9f3407626839da27470ab89da Author: Danny Chan AuthorDate: Thu Dec 1 17:13:59 2022 +0800

[hudi] 04/17: [MINOR] Disable the `SparkSqlCoreFlow` tests (#7368)

2022-12-13 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch release-0.12.2 in repository https://gitbox.apache.org/repos/asf/hudi.git commit 93d766df0c45d4547249a34f93b345a246f8107a Author: Jon Vexler AuthorDate: Fri Dec 2 22:16:46 2022 -0500

[hudi] 15/17: [HUDI-5163] Fix failure handling with spark datasource write (#7140)

2022-12-13 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch release-0.12.2 in repository https://gitbox.apache.org/repos/asf/hudi.git commit d3f420d99485ce4ce03b047ca074e086fb3875b0 Author: Sivabalan Narayanan AuthorDate: Wed Dec 7 09:17:27 2022 -0800

[hudi] 10/17: [HUDI-3661] Flink async compaction is not thread safe when use watermark (#7399)

2022-12-13 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch release-0.12.2 in repository https://gitbox.apache.org/repos/asf/hudi.git commit bd60b7954fbeaf037241f96c01dcc42678180bb4 Author: Danny Chan AuthorDate: Wed Dec 7 18:31:26 2022 +0800

[hudi] 07/17: [HUDI-5294] Support type change for schema on read + reconcile schema (#7326)

2022-12-13 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch release-0.12.2 in repository https://gitbox.apache.org/repos/asf/hudi.git commit 421f1cd6b75b3a9599edd0953273f606ac112e3d Author: xiarixiaoyao AuthorDate: Wed Dec 7 10:38:54 2022 +0800

[hudi] 09/17: [HUDI-5334] Fix checkpoint reading for structured streaming (#7389)

2022-12-13 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch release-0.12.2 in repository https://gitbox.apache.org/repos/asf/hudi.git commit bc6d6aa35d3da3f3d5a56891b922139cea7565c8 Author: Shiyan Xu <2701446+xushi...@users.noreply.github.com> AuthorDate:

  1   2   3   4   5   >