Re: [PR] [HUDI-6924] Fix hoodie table config not wok in table properties [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9836: URL: https://github.com/apache/hudi/pull/9836#issuecomment-1752661843 ## CI report: * ec64b99d8525d88a1844e434000d83d06e74fe75 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [MINOR] scanner should closed properly in HoodieComactor [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9835: URL: https://github.com/apache/hudi/pull/9835#issuecomment-1752661744 ## CI report: * 0e564fbc0b7e5e10e6a61bff77014de0f33b82bb Azure:

Re: [I] [SUPPORT] Writing Hudi tables with Flink fails with HFile exceptions [hudi]

2023-10-09 Thread via GitHub
galadrielwithlaptop commented on issue #9495: URL: https://github.com/apache/hudi/issues/9495#issuecomment-1752659628 A dstream job which writes data in hudi format to ABFS directory fails saying that “Caused by: java.lang.NoClassDefFoundError: Could not initialize class

[jira] [Created] (HUDI-6925) Do not list all partitions when deleting partition

2023-10-09 Thread Qijun Fu (Jira)
Qijun Fu created HUDI-6925: -- Summary: Do not list all partitions when deleting partition Key: HUDI-6925 URL: https://issues.apache.org/jira/browse/HUDI-6925 Project: Apache Hudi Issue Type:

[jira] [Updated] (HUDI-6381) Bucket bulk insert should support CoW table

2023-10-09 Thread Qijun Fu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qijun Fu updated HUDI-6381: --- Fix Version/s: 0.14.0 > Bucket bulk insert should support CoW table >

[jira] [Resolved] (HUDI-6381) Bucket bulk insert should support CoW table

2023-10-09 Thread Qijun Fu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qijun Fu resolved HUDI-6381. > Bucket bulk insert should support CoW table > --- > >

[jira] [Resolved] (HUDI-6534) Spark Consistent Hashing row writer support

2023-10-09 Thread Qijun Fu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qijun Fu resolved HUDI-6534. > Spark Consistent Hashing row writer support > --- > >

Re: [I] [BUG] S3 Deltastreamer: Block has already been inflated [hudi]

2023-10-09 Thread via GitHub
ad1happy2go commented on issue #6428: URL: https://github.com/apache/hudi/issues/6428#issuecomment-1752649988 @envomp Are you setting `fs.s3a.connection.maximum` to a higher value. That might fix the Connection timeout issue. -- This is an automated message from the Apache Git Service.

[jira] [Closed] (HUDI-6906) Timeline based checkpoint metadata should not rely on file system ckp files

2023-10-09 Thread Qijun Fu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qijun Fu closed HUDI-6906. -- Resolution: Duplicate > Timeline based checkpoint metadata should not rely on file system ckp files >

[jira] [Closed] (HUDI-6905) Timeline based checkpoint metadata should not rely on file system ckp files

2023-10-09 Thread Qijun Fu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qijun Fu closed HUDI-6905. -- Resolution: Duplicate > Timeline based checkpoint metadata should not rely on file system ckp files >

Re: [PR] [MINOR] scanner should closed properly in HoodieComactor [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9835: URL: https://github.com/apache/hudi/pull/9835#issuecomment-1752647098 ## CI report: * 0e564fbc0b7e5e10e6a61bff77014de0f33b82bb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9581: URL: https://github.com/apache/hudi/pull/9581#issuecomment-1752646021 ## CI report: * 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN * bda7b26fbc477b408e8658eb6e52badeb5c653e0 Azure:

Re: [I] [SUPPORT] I would like to use Hudi from the Julia language in streaming fashion [hudi]

2023-10-09 Thread via GitHub
ad1happy2go commented on issue #9735: URL: https://github.com/apache/hudi/issues/9735#issuecomment-1752642166 @schlichtanders Closing this issue. Please reopen in case of any concerns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] [SUPPORT] I would like to use Hudi from the Julia language in streaming fashion [hudi]

2023-10-09 Thread via GitHub
codope closed issue #9735: [SUPPORT] I would like to use Hudi from the Julia language in streaming fashion URL: https://github.com/apache/hudi/issues/9735 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[jira] [Updated] (HUDI-6924) Fix hoodie table config not wok in table properties

2023-10-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6924: - Labels: pull-request-available (was: ) > Fix hoodie table config not wok in table properties >

[PR] [HUDI-6924] Fix hoodie table config not wok in table properties [hudi]

2023-10-09 Thread via GitHub
wecharyu opened a new pull request, #9836: URL: https://github.com/apache/hudi/pull/9836 ### Change Logs Fix the bug that hoodie table config not work in table properties. Including: 1. Combine `mapTableConfigsToSqlOptions` and `mapDataSourceWriteOptionsToSqlOptions` to

Re: [I] [SUPPORT] Spark job stuck after completion, due to some non daemon threads still running [hudi]

2023-10-09 Thread via GitHub
pravin1406 commented on issue #9826: URL: https://github.com/apache/hudi/issues/9826#issuecomment-1752626251 @yihua anything on this ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[jira] [Created] (HUDI-6924) Fix hoodie table config not wok in table properties

2023-10-09 Thread Wechar (Jira)
Wechar created HUDI-6924: Summary: Fix hoodie table config not wok in table properties Key: HUDI-6924 URL: https://issues.apache.org/jira/browse/HUDI-6924 Project: Apache Hudi Issue Type: Bug

[PR] [MINOR] scanner should closed properly in HoodieComactor [hudi]

2023-10-09 Thread via GitHub
xuzifu666 opened a new pull request, #9835: URL: https://github.com/apache/hudi/pull/9835 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1350025100 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java: ## @@ -16,27 +16,33 @@ * limitations under the License. */ -package

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1350024714 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/upgrade/TestUpgradeDowngrade.java: ## @@ -844,6 +847,47 @@ private Pair, List>

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1350023322 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/plan/generators/BaseHoodieCompactionPlanGenerator.java: ## @@ -118,8 +120,21 @@ public

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1350022791 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java: ## @@ -726,19 +727,24 @@ protected void

[jira] [Updated] (HUDI-6916) Fix excessive object creation in custom key generator

2023-10-09 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-6916: -- Fix Version/s: 0.14.1 > Fix excessive object creation in custom key generator >

[jira] [Closed] (HUDI-6916) Fix excessive object creation in custom key generator

2023-10-09 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-6916. - Resolution: Fixed > Fix excessive object creation in custom key generator >

[hudi] branch master updated: [HUDI-6916] Improve performance of Custom Key Generators (#9821)

2023-10-09 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 1f1c25c4d86 [HUDI-6916] Improve performance of

Re: [PR] [HUDI-6916] Improve performance of Custom Key Generators [hudi]

2023-10-09 Thread via GitHub
codope merged PR #9821: URL: https://github.com/apache/hudi/pull/9821 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-6923] Fixing bug with sanitization for rowSource [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9834: URL: https://github.com/apache/hudi/pull/9834#issuecomment-1752569635 ## CI report: * bd94b2cc68c5c339394bcaf9b077093eb6e9f18e Azure:

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9581: URL: https://github.com/apache/hudi/pull/9581#issuecomment-1752568707 ## CI report: * 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN * 4ce9d07dea4d5ab3088037461f9fa3b884b64eb6 Azure:

Re: [I] [BUG] S3 Deltastreamer: Block has already been inflated [hudi]

2023-10-09 Thread via GitHub
envomp commented on issue #6428: URL: https://github.com/apache/hudi/issues/6428#issuecomment-1752559828 Hey folks, This issue: https://gist.github.com/envomp/268bdd35a3b3399db59583c0e159c229#file-cover-logs Seems to be a cover-up to real underlying issue which in our case

[jira] [Updated] (HUDI-6923) Fix sanitisation of fields for rowSource with RowBasedSchemaProvider

2023-10-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6923: - Labels: pull-request-available (was: ) > Fix sanitisation of fields for rowSource with

Re: [PR] [HUDI-6923] Fixing bug with sanitization for rowSource [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9834: URL: https://github.com/apache/hudi/pull/9834#issuecomment-1752555349 ## CI report: * bd94b2cc68c5c339394bcaf9b077093eb6e9f18e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9581: URL: https://github.com/apache/hudi/pull/9581#issuecomment-1752554282 ## CI report: * 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN * 4ce9d07dea4d5ab3088037461f9fa3b884b64eb6 Azure:

Re: [PR] [MINOR] Flink Coordinator stop current instant heartbeat before create a new instant [hudi]

2023-10-09 Thread via GitHub
stream2000 closed pull request #9175: [MINOR] Flink Coordinator stop current instant heartbeat before create a new instant URL: https://github.com/apache/hudi/pull/9175 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9581: URL: https://github.com/apache/hudi/pull/9581#issuecomment-1752540717 ## CI report: * 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN * 4ce9d07dea4d5ab3088037461f9fa3b884b64eb6 Azure:

[jira] [Updated] (HUDI-6923) Fix sanitisation of fields for rowSource with RowBasedSchemaProvider

2023-10-09 Thread Harshal Patil (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harshal Patil updated HUDI-6923: Description: `RowSource` provider schemaProvider as  RowBasedSchemaProvider . Which can be

[jira] [Created] (HUDI-6923) Fix sanitisation of fields for rowSource with RowBasedSchemaProvider

2023-10-09 Thread Harshal Patil (Jira)
Harshal Patil created HUDI-6923: --- Summary: Fix sanitisation of fields for rowSource with RowBasedSchemaProvider Key: HUDI-6923 URL: https://issues.apache.org/jira/browse/HUDI-6923 Project: Apache Hudi

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349974150 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -216,6 +233,9 @@ protected List buildFileGroups(Stream baseFileS

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
codope commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1349972699 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/rollback/TestMergeOnReadRollbackActionExecutor.java: ## @@ -126,21 +126,19 @@ public void

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
codope commented on PR #9581: URL: https://github.com/apache/hudi/pull/9581#issuecomment-1752515071 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[PR] Fixing bug with sanitization for rowSource [hudi]

2023-10-09 Thread via GitHub
harsh1231 opened a new pull request, #9834: URL: https://github.com/apache/hudi/pull/9834 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any

Re: [PR] [HUDI-6873] fix clustering mor [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9774: URL: https://github.com/apache/hudi/pull/9774#issuecomment-1752481582 ## CI report: * 658c987d20c827851b05cd530d3adfda5038df7c Azure:

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9581: URL: https://github.com/apache/hudi/pull/9581#issuecomment-1752481011 ## CI report: * 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN * 1a11ff678d2345105879a6faa951c18d94dfa1ba Azure:

Re: [PR] [HUDI-6873] fix clustering mor [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9774: URL: https://github.com/apache/hudi/pull/9774#issuecomment-1752470130 ## CI report: * 658c987d20c827851b05cd530d3adfda5038df7c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
codope commented on code in PR #9581: URL: https://github.com/apache/hudi/pull/9581#discussion_r1349937000 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java: ## @@ -2029,6 +2029,10 @@ public long getLogFileDataBlockMaxSize() {

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
codope commented on code in PR #9581: URL: https://github.com/apache/hudi/pull/9581#discussion_r1349936513 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDeleteBlock.java: ## @@ -65,17 +69,44 @@ public class HoodieDeleteBlock extends HoodieLogBlock {

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
codope commented on code in PR #9581: URL: https://github.com/apache/hudi/pull/9581#discussion_r1349935956 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java: ## @@ -173,18 +173,18 @@ public static HoodieRecord tagRecord(HoodieRecord

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
codope commented on code in PR #9581: URL: https://github.com/apache/hudi/pull/9581#discussion_r1349935738 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java: ## @@ -752,6 +752,14 @@ public class HoodieWriteConfig extends HoodieConfig

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
codope commented on code in PR #9581: URL: https://github.com/apache/hudi/pull/9581#discussion_r1349935195 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/ListBasedHoodieBloomIndexHelper.java: ## @@ -64,22 +64,23 @@ public HoodiePairData

Re: [I] [SUPPORT] Clarify the Multi Writer guarantee for duplicates [hudi]

2023-10-09 Thread via GitHub
xicm commented on issue #9831: URL: https://github.com/apache/hudi/issues/9831#issuecomment-1752426819 The problem you described is correct and multi-writer does have this problem now. Danny is working for a Non blocking concurrency control https://github.com/apache/hudi/pull/7907 --

<    1   2