Re: [PR] [HUDI-6872] Simplify Out Of Box Schema Evolution Functionality [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1803296619 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * e32b58f7ce1880568566be0c8a6940ae2f3a1016 UNKNOWN * d3e58fee4a8cfb636419d9eeb50fa2480b56e2c1 Azure:

Re: [PR] [HUDI-6872][DNM] Simplify Out Of Box Schema Evolution Functionality (test PR to check CI) [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10028: URL: https://github.com/apache/hudi/pull/10028#issuecomment-1803289243 ## CI report: * 4c36398beb08a1eb91959ce151ebf8fd159bb8d6 Azure:

Re: [PR] [HUDI-7054][FOLLOW_UP] HoodieCatalogTable should ignore lazily deleted partitions [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10024: URL: https://github.com/apache/hudi/pull/10024#issuecomment-1803289144 ## CI report: * 17566af6a57a43de3b2a5d5ab7bcd8bd28336307 Azure:

Re: [PR] [HUDI-6872] Simplify Out Of Box Schema Evolution Functionality [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1803288697 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * e32b58f7ce1880568566be0c8a6940ae2f3a1016 UNKNOWN * d3e58fee4a8cfb636419d9eeb50fa2480b56e2c1 Azure:

Re: [PR] [HUDI-7054][FOLLOW_UP] HoodieCatalogTable should ignore lazily deleted partitions [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10024: URL: https://github.com/apache/hudi/pull/10024#issuecomment-1803281654 ## CI report: * 17566af6a57a43de3b2a5d5ab7bcd8bd28336307 Azure:

Re: [PR] [HUDI-6872][DNM] Simplify Out Of Box Schema Evolution Functionality (test PR to check CI) [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10028: URL: https://github.com/apache/hudi/pull/10028#issuecomment-1803281756 ## CI report: * 91a7b8640540e841522e33c001fc8c0f4abddb4a Azure:

[jira] [Closed] (HUDI-6992) IncrementalInputSplits incorrectly set the latestCommit attr

2023-11-08 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-6992. Resolution: Fixed Fixed via master branch: 44ca2bbcfd1512a55155e1033a9c9aca132efae6 >

(hudi) branch master updated: [HUDI-6992] IncrementalInputSplits incorrectly set the latestCommit attribute (#9923)

2023-11-08 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 44ca2bbcfd1 [HUDI-6992] IncrementalInputSplits

Re: [PR] [HUDI-6992] IncrementalInputSplits incorrectly set the latestCommit attr [hudi]

2023-11-08 Thread via GitHub
danny0405 merged PR #9923: URL: https://github.com/apache/hudi/pull/9923 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]

2023-11-08 Thread via GitHub
ad1happy2go commented on issue #10017: URL: https://github.com/apache/hudi/issues/10017#issuecomment-1803263076 @zdl1 Can you also share sample code/data what you are trying if possible. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] [HUDI-7057] Support CopyToTableProcedure with patitial column copy [hudi]

2023-11-08 Thread via GitHub
xuzifu666 commented on PR #10025: URL: https://github.com/apache/hudi/pull/10025#issuecomment-1803255723 CI seems well done @danny0405 PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [HUDI-7054][FOLLOW_UP] HoodieCatalogTable should ignore lazily deleted partitions [hudi]

2023-11-08 Thread via GitHub
boneanxs commented on code in PR #10024: URL: https://github.com/apache/hudi/pull/10024#discussion_r1387540620 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala: ## @@ -169,9 +170,10 @@ class

Re: [PR] [HUDI-6872][DNM] Simplify Out Of Box Schema Evolution Functionality (test PR to check CI) [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10028: URL: https://github.com/apache/hudi/pull/10028#issuecomment-1803229404 ## CI report: * 91a7b8640540e841522e33c001fc8c0f4abddb4a Azure:

Re: [PR] [HUDI-7057] Support CopyToTableProcedure with patitial column copy [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10025: URL: https://github.com/apache/hudi/pull/10025#issuecomment-1803229350 ## CI report: * 80a394967d09baac64231af055c40996dbb2a7fd Azure:

Re: [PR] [HUDI-7053] Fix the filter pushdown logic for file group reader [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10018: URL: https://github.com/apache/hudi/pull/10018#issuecomment-1803229313 ## CI report: * e420cdf46604699ae6587b9aa13e6e9c0d139d38 Azure:

Re: [PR] [MINOR] Change some default configs for 1.0.0-beta [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1803229209 ## CI report: * bb197bc8649a9b8dfcd41d9ed3e4af8e0afbeb9b Azure:

Re: [PR] [HUDI-6872] Simplify Out Of Box Schema Evolution Functionality [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1803228898 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * e32b58f7ce1880568566be0c8a6940ae2f3a1016 UNKNOWN * d3e58fee4a8cfb636419d9eeb50fa2480b56e2c1 Azure:

Re: [PR] [HUDI-7054][FOLLOW_UP] HoodieCatalogTable should ignore lazily deleted partitions [hudi]

2023-11-08 Thread via GitHub
danny0405 commented on code in PR #10024: URL: https://github.com/apache/hudi/pull/10024#discussion_r1387533789 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/catalog/HoodieCatalogTable.scala: ## @@ -169,9 +170,10 @@ class

Re: [PR] [HUDI-7054][FOLLOW_UP] HoodieCatalogTable should ignore lazily deleted partitions [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10024: URL: https://github.com/apache/hudi/pull/10024#issuecomment-1803194775 ## CI report: * 17566af6a57a43de3b2a5d5ab7bcd8bd28336307 Azure:

Re: [PR] [MINOR] Change some default configs for 1.0.0-beta [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1803194686 ## CI report: * 7e450aee63b81c2d28d04d927eadf5ca006e8a19 Azure:

Re: [PR] [MINOR] Change some default configs for 1.0.0-beta [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1803187055 ## CI report: * 7e450aee63b81c2d28d04d927eadf5ca006e8a19 Azure:

Re: [PR] [HUDI-6872][DNM] Simplify Out Of Box Schema Evolution Functionality (test PR to check CI) [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10028: URL: https://github.com/apache/hudi/pull/10028#issuecomment-1803180699 ## CI report: * 91a7b8640540e841522e33c001fc8c0f4abddb4a Azure:

Re: [PR] [HUDI-7047] Fix Stack Overflow Caused by partition pruning [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10026: URL: https://github.com/apache/hudi/pull/10026#issuecomment-1803180657 ## CI report: * ae33aa0b78cb61bcf6c834bfd404d0c7296ac05b Azure:

Re: [PR] [HUDI-7044] Skip reading records for delete blocks for positional merging [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10005: URL: https://github.com/apache/hudi/pull/10005#issuecomment-1803180577 ## CI report: * baaf10ac8ac319fb3e776b33b4386755bf034cb6 Azure:

Re: [PR] [HUDI-6872] Simplify Out Of Box Schema Evolution Functionality [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1803180312 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * e32b58f7ce1880568566be0c8a6940ae2f3a1016 UNKNOWN * db06d7f6394d26bec65a629ed2b567754d28a46a Azure:

Re: [PR] [HUDI-6993] Support Flink 1.18 [hudi]

2023-11-08 Thread via GitHub
PrabhuJoseph commented on PR #9949: URL: https://github.com/apache/hudi/pull/9949#issuecomment-1803166707 Thanks @danny0405 for the review and commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] [SUPPORT] DBT Merge creates duplicates [hudi]

2023-11-08 Thread via GitHub
faizhasan commented on issue #7244: URL: https://github.com/apache/hudi/issues/7244#issuecomment-1803162436 Hi @amrishlal apologies for the delay. I was able to test this and saw the following behavior. with dbt 1.6.2, dbt-spark adapter to execute models on thriftserver: -

Re: [PR] [HUDI-6872][DNM] Simplify Out Of Box Schema Evolution Functionality (test PR to check CI) [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10028: URL: https://github.com/apache/hudi/pull/10028#issuecomment-1803152853 ## CI report: * 91a7b8640540e841522e33c001fc8c0f4abddb4a Azure:

Re: [PR] [HUDI-6872] Simplify Out Of Box Schema Evolution Functionality [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1803152527 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * e32b58f7ce1880568566be0c8a6940ae2f3a1016 UNKNOWN * db06d7f6394d26bec65a629ed2b567754d28a46a Azure:

Re: [PR] [HUDI-6872] Simplify Out Of Box Schema Evolution Functionality [hudi]

2023-11-08 Thread via GitHub
nsivabalan commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1803150479 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [HUDI-6872][DNM] Simplify Out Of Box Schema Evolution Functionality (test PR to check CI) [hudi]

2023-11-08 Thread via GitHub
nsivabalan commented on PR #10028: URL: https://github.com/apache/hudi/pull/10028#issuecomment-1803150379 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [HUDI-7058] check if option is empty before get in HoodieBaseFileGroupRecordBuffer [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10027: URL: https://github.com/apache/hudi/pull/10027#issuecomment-1803148099 ## CI report: * 384dbfa4bc44704f0151f9fa1330cbcff172d677 Azure:

Re: [PR] [HUDI-6872][DNM] Simplify Out Of Box Schema Evolution Functionality (test PR to check CI) [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10028: URL: https://github.com/apache/hudi/pull/10028#issuecomment-1803148139 ## CI report: * 91a7b8640540e841522e33c001fc8c0f4abddb4a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-7047] Fix Stack Overflow Caused by partition pruning [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10026: URL: https://github.com/apache/hudi/pull/10026#issuecomment-1803148073 ## CI report: * ae33aa0b78cb61bcf6c834bfd404d0c7296ac05b Azure:

Re: [PR] [HUDI-6872] Simplify Out Of Box Schema Evolution Functionality [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1803147734 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * e32b58f7ce1880568566be0c8a6940ae2f3a1016 UNKNOWN * db06d7f6394d26bec65a629ed2b567754d28a46a Azure:

Re: [PR] [HUDI-7058] check if option is empty before get in HoodieBaseFileGroupRecordBuffer [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10027: URL: https://github.com/apache/hudi/pull/10027#issuecomment-1803143323 ## CI report: * 384dbfa4bc44704f0151f9fa1330cbcff172d677 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-7047] Fix Stack Overflow Caused by partition pruning [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10026: URL: https://github.com/apache/hudi/pull/10026#issuecomment-1803143273 ## CI report: * ae33aa0b78cb61bcf6c834bfd404d0c7296ac05b UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-6872][DNM] Simplify Out Of Box Schema Evolution Functionality (test PR to check CI) [hudi]

2023-11-08 Thread via GitHub
nsivabalan commented on PR #10028: URL: https://github.com/apache/hudi/pull/10028#issuecomment-1803142917 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] [HUDI-6872][DNM] Simplify Out Of Box Schema Evolution Functionality [hudi]

2023-11-08 Thread via GitHub
nsivabalan opened a new pull request, #10028: URL: https://github.com/apache/hudi/pull/10028 ### Change Logs Simplify Out Of Box Schema Evolution Functionality ### Impact - Simplify Out Of Box Schema Evolution Functionality ### Risk level (write none, low medium

[jira] [Updated] (HUDI-7058) HoodieBaseFileGroupRecordBuffer doesn't check if option is empty

2023-11-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7058: - Labels: pull-request-available (was: ) > HoodieBaseFileGroupRecordBuffer doesn't check if option

[PR] [HUDI-7058] check if option is empty before get in HoodieBaseFileGroupRecordBuffer [hudi]

2023-11-08 Thread via GitHub
jonvex opened a new pull request, #10027: URL: https://github.com/apache/hudi/pull/10027 ### Change Logs HoodieBaseFileGroupRecordBuffer does a record merge and then gets the option without checking. testBaseFileAndLogFileUpdateMatchesDeleteBlock exposes that the option can be

[jira] [Created] (HUDI-7058) HoodieBaseFileGroupRecordBuffer doesn't check if option is empty

2023-11-08 Thread Jonathan Vexler (Jira)
Jonathan Vexler created HUDI-7058: - Summary: HoodieBaseFileGroupRecordBuffer doesn't check if option is empty Key: HUDI-7058 URL: https://issues.apache.org/jira/browse/HUDI-7058 Project: Apache Hudi

[PR] [HUDI-7047] Fix Stack Overflow Caused by partition pruning [hudi]

2023-11-08 Thread via GitHub
jonvex opened a new pull request, #10026: URL: https://github.com/apache/hudi/pull/10026 ### Change Logs hasPushedDownPartitionPredicates is not set to true in all cases when listFiles is called, so the planner repeatedly tries to resolve it ### Impact no more stack

Re: [I] UPSERTs are taking time [hudi]

2023-11-08 Thread via GitHub
nsivabalan commented on issue #9976: URL: https://github.com/apache/hudi/issues/9976#issuecomment-1803119793 and yeah. upgrading to 0.14.0, you can leverage RLI and that should def boost your index latencies and write latencies. -- This is an automated message from the Apache Git

Re: [I] UPSERTs are taking time [hudi]

2023-11-08 Thread via GitHub
nsivabalan commented on issue #9976: URL: https://github.com/apache/hudi/issues/9976#issuecomment-1803119446 yeah. As I suggested before, you may want to try our MOR table. and try using SIMPLE index. in 0.10.1 hudi uses bloom index and for random keys it might incur some unnecessary

Re: [PR] [HUDI-7057] Support CopyToTableProcedure with patitial column copy [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10025: URL: https://github.com/apache/hudi/pull/10025#issuecomment-1803111282 ## CI report: * 80a394967d09baac64231af055c40996dbb2a7fd Azure:

Re: [PR] [HUDI-7053] Fix the filter pushdown logic for file group reader [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10018: URL: https://github.com/apache/hudi/pull/10018#issuecomment-1803111222 ## CI report: * c5812540396b56db64df779ab7147cc0cede626a Azure:

(hudi) branch master updated: [HUDI-7056] Add config for merging using positions (#10022)

2023-11-08 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 2e2e39377e2 [HUDI-7056] Add config for merging

Re: [PR] [HUDI-7056] Add config for merging using positions [hudi]

2023-11-08 Thread via GitHub
yihua merged PR #10022: URL: https://github.com/apache/hudi/pull/10022 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]

2023-11-08 Thread via GitHub
zdl1 commented on issue #10017: URL: https://github.com/apache/hudi/issues/10017#issuecomment-1803107881 > Are the inflight and requested compaction metdata files empty also? The inflight file is empty but the requested file is not. -- This is an automated message from the

Re: [PR] [HUDI-7057] Support CopyToTableProcedure with patitial column copy [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10025: URL: https://github.com/apache/hudi/pull/10025#issuecomment-1803106701 ## CI report: * 80a394967d09baac64231af055c40996dbb2a7fd UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-7053] Fix the filter pushdown logic for file group reader [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10018: URL: https://github.com/apache/hudi/pull/10018#issuecomment-1803106634 ## CI report: * c5812540396b56db64df779ab7147cc0cede626a Azure:

Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]

2023-11-08 Thread via GitHub
danny0405 commented on issue #10017: URL: https://github.com/apache/hudi/issues/10017#issuecomment-1803103939 Are the inflight and requested compaction metdata files empty also? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] [HUDI-7050]Flink HoodieHiveCatalog supports hadoop parameters [hudi]

2023-11-08 Thread via GitHub
danny0405 commented on code in PR #10013: URL: https://github.com/apache/hudi/pull/10013#discussion_r1387440082 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieCatalogUtil.java: ## @@ -61,9 +61,10 @@ public class HoodieCatalogUtil { *

Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]

2023-11-08 Thread via GitHub
zdl1 commented on issue #10017: URL: https://github.com/apache/hudi/issues/10017#issuecomment-1803102881 > what release did you use? Thanks for your information! But now I am using flink1.14 and hudi0.13.1 -- This is an automated message from the Apache Git Service. To respond

Re: [PR] [HUDI-7056] Add config for merging using positions [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10022: URL: https://github.com/apache/hudi/pull/10022#issuecomment-1803101905 ## CI report: * 8f150c02d9ff127c3a0dd0ec06d46c696de88a70 Azure:

Re: [PR] [HUDI-7050]Flink HoodieHiveCatalog supports hadoop parameters [hudi]

2023-11-08 Thread via GitHub
danny0405 commented on code in PR #10013: URL: https://github.com/apache/hudi/pull/10013#discussion_r1387438530 ## hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieCatalogFactory.java: ## @@ -40,6 +40,7 @@ public class HoodieCatalogFactory

Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]

2023-11-08 Thread via GitHub
danny0405 commented on issue #10017: URL: https://github.com/apache/hudi/issues/10017#issuecomment-1803093991 what release did you use? Since 0.14.0, for MOR + INSERT, we write parquets directly, you should use clustering instead of compaction. -- This is an automated message from the

Re: [I] [SUPPORT] Possible memory leak issue for org.apache.hadoop.hive.conf.HiveConf while using Flink into Hudi [hudi]

2023-11-08 Thread via GitHub
danny0405 commented on issue #10023: URL: https://github.com/apache/hudi/issues/10023#issuecomment-1803090137 There is an known Hive conf resource leak fix: https://github.com/apache/hudi/pull/8050/files, but the 0.13.0 should include this fix. -- This is an automated message from the

Re: [I] [SUPPORT] Solution for synchronizing the entire database table in flink [hudi]

2023-11-08 Thread via GitHub
danny0405 commented on issue #9965: URL: https://github.com/apache/hudi/issues/9965#issuecomment-1803084529 yeah, already on the list. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] [BUG] Spark will read invalid timestamp(3) data when record in log is older than the same in parquet. [hudi]

2023-11-08 Thread via GitHub
danny0405 commented on issue #10012: URL: https://github.com/apache/hudi/issues/10012#issuecomment-1803082627 Looks like the avro schema is using `timestamp-millis` as the logical data type, so in `EventTimeAvroPayload`, did you debug a little bit where the `timestamp-micros` come from?

[jira] [Updated] (HUDI-7057) Support CopyToTableProcedure with patitial column copy

2023-11-08 Thread xy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xy updated HUDI-7057: - Description: currently CopyToTableProcedure only support all schema copy to a new table,but many sences user need a part

[jira] [Closed] (HUDI-7017) Prevent full schema evolution from wrongly falling back to OOB

2023-11-08 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-7017. Resolution: Fixed Fixed via master branch: d149d787e9d3dbf425c0dd5ca0265bed5fe2795f > Prevent full schema

(hudi) branch master updated: [HUDI-7055] Support reading only log files in file group reader-based Spark parquet file format (#10020)

2023-11-08 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 530640f61a5 [HUDI-7055] Support reading only log

Re: [PR] [HUDI-7055] Support reading only log files in file group reader-based Spark parquet file format [hudi]

2023-11-08 Thread via GitHub
yihua merged PR #10020: URL: https://github.com/apache/hudi/pull/10020 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Closed] (HUDI-6993) Support Flink 1.18

2023-11-08 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-6993. Resolution: Fixed Fixed via master branch: bd136addc24f4d03608f5069b98c2748e463169f > Support Flink 1.18 >

[jira] [Updated] (HUDI-7017) Prevent full schema evolution from wrongly falling back to OOB

2023-11-08 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-7017: - Fix Version/s: 1.0.0 > Prevent full schema evolution from wrongly falling back to OOB >

(hudi) branch master updated: [HUDI-7017] Prevent full schema evolution from wrongly falling back to OOB schema evolution (#9966)

2023-11-08 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new d149d787e9d [HUDI-7017] Prevent full schema

Re: [PR] [HUDI-7017] Prevent full schema evolution from wrongly falling back t… [hudi]

2023-11-08 Thread via GitHub
danny0405 merged PR #9966: URL: https://github.com/apache/hudi/pull/9966 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Updated] (HUDI-6993) Support Flink 1.18

2023-11-08 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6993: - Fix Version/s: 1.0.0 > Support Flink 1.18 > -- > > Key: HUDI-6993 >

Re: [PR] [HUDI-7017] Prevent full schema evolution from wrongly falling back t… [hudi]

2023-11-08 Thread via GitHub
danny0405 commented on PR #9966: URL: https://github.com/apache/hudi/pull/9966#issuecomment-1803076235 Tests passed: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=20721=results -- This is an automated message from the Apache Git Service. To respond to the

(hudi) branch master updated (7c17964fe73 -> bd136addc24)

2023-11-08 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 7c17964fe73 [HUDI-6495][RFC-66] Non-blocking Concurrency Control (#7907) add bd136addc24 [HUDI-6993] Support

Re: [PR] [HUDI-6993] Support Flink 1.18 [hudi]

2023-11-08 Thread via GitHub
danny0405 merged PR #9949: URL: https://github.com/apache/hudi/pull/9949 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Updated] (HUDI-7057) Support CopyToTableProcedure with patitial column copy

2023-11-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7057: - Labels: pull-request-available sparksql (was: sparksql) > Support CopyToTableProcedure with

Re: [PR] [HUDI-6993] Support Flink 1.18 [hudi]

2023-11-08 Thread via GitHub
danny0405 commented on PR #9949: URL: https://github.com/apache/hudi/pull/9949#issuecomment-1803074942 Tests passed: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=20702=results -- This is an automated message from the Apache Git Service. To respond to the

[PR] [HUDI-7057] Support CopyToTableProcedure with patitial column copy [hudi]

2023-11-08 Thread via GitHub
xuzifu666 opened a new pull request, #10025: URL: https://github.com/apache/hudi/pull/10025 ### Change Logs currently CopyToTableProcedure only support all schema copy to a new table,but many sences user need a part columns from table schema ### Impact none ###

[jira] [Updated] (HUDI-7057) Support CopyToTableProcedure with patitial column copy

2023-11-08 Thread xy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xy updated HUDI-7057: - Labels: sparksql (was: ) > Support CopyToTableProcedure with patitial column copy >

[jira] [Created] (HUDI-7057) Support CopyToTableProcedure with patitial column copy

2023-11-08 Thread xy (Jira)
xy created HUDI-7057: Summary: Support CopyToTableProcedure with patitial column copy Key: HUDI-7057 URL: https://issues.apache.org/jira/browse/HUDI-7057 Project: Apache Hudi Issue Type: Improvement

[jira] [Assigned] (HUDI-7057) Support CopyToTableProcedure with patitial column copy

2023-11-08 Thread xy (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xy reassigned HUDI-7057: Assignee: xy > Support CopyToTableProcedure with patitial column copy >

Re: [PR] [HUDI-6806] Support Spark 3.5.0 [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9717: URL: https://github.com/apache/hudi/pull/9717#issuecomment-1803071474 ## CI report: * 9b8fdd2d1b69da528069e364790b53af1d6150af UNKNOWN * 43253eac6f27abbb614bb80d514d5ea3e30e09a1 Azure:

Re: [PR] [HUDI-7054][FOLLOW_UP] HoodieCatalogTable should ignore lazily deleted partitions [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10024: URL: https://github.com/apache/hudi/pull/10024#issuecomment-1803071823 ## CI report: * 17566af6a57a43de3b2a5d5ab7bcd8bd28336307 Azure:

Re: [I] [SUPPORT] Some .commit files generated by compaction with 0 size [hudi]

2023-11-08 Thread via GitHub
zdl1 commented on issue #10017: URL: https://github.com/apache/hudi/issues/10017#issuecomment-1803070116 Thanks for your help, and there is another question, for an existing partition, the new insertion record to an existing bucket will be marked as **numUpdateWrites**, so how could I

Re: [PR] [HUDI-7054][FOLLOW_UP] HoodieCatalogTable should ignore lazily deleted partitions [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10024: URL: https://github.com/apache/hudi/pull/10024#issuecomment-1803066773 ## CI report: * 17566af6a57a43de3b2a5d5ab7bcd8bd28336307 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-7053] Fix the filter pushdown logic for file group reader [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10018: URL: https://github.com/apache/hudi/pull/10018#issuecomment-1803066730 ## CI report: * c5812540396b56db64df779ab7147cc0cede626a Azure:

Re: [PR] [HUDI-6495][RFC-66] Non-blocking Concurrency Control [hudi]

2023-11-08 Thread via GitHub
vinothchandar merged PR #7907: URL: https://github.com/apache/hudi/pull/7907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(hudi) branch master updated (43a39b907bc -> 7c17964fe73)

2023-11-08 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 43a39b907bc ShowPartitionsCommand should consider lazy delete_partitions (#10019) add 7c17964fe73

Re: [PR] [HUDI-7055] Support reading only log files in file group reader-based Spark parquet file format [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10020: URL: https://github.com/apache/hudi/pull/10020#issuecomment-1803061504 ## CI report: * 8b39f72f16c9b708667dc17ee51875cf3c9b7364 Azure:

Re: [PR] [HUDI-7053] Fix the filter pushdown logic for file group reader [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10018: URL: https://github.com/apache/hudi/pull/10018#issuecomment-1803061463 ## CI report: * c5812540396b56db64df779ab7147cc0cede626a Azure:

Re: [PR] [HUDI-6872] Simplify Out Of Box Schema Evolution Functionality [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1803061117 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * e32b58f7ce1880568566be0c8a6940ae2f3a1016 UNKNOWN * db06d7f6394d26bec65a629ed2b567754d28a46a Azure:

[PR] [HUDI-7054][FOLLOW_UP] HoodieCatalogTable should ignore lazily deleted partitions [hudi]

2023-11-08 Thread via GitHub
boneanxs opened a new pull request, #10024: URL: https://github.com/apache/hudi/pull/10024 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ Thinking it might be better to directly let `HoodieCatalogTable` to filter deleted

Re: [PR] [HUDI-6508] Fix compile errors with JDK11 [hudi]

2023-11-08 Thread via GitHub
Zouxxyy commented on PR #9300: URL: https://github.com/apache/hudi/pull/9300#issuecomment-1803034364 > @Zouxxyy : Can you fix the merge conflicts ? done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [MINOR] Change some default configs for 1.0.0-beta [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1803028334 ## CI report: * 7e450aee63b81c2d28d04d927eadf5ca006e8a19 Azure:

[I] [SUPPORT] Possible memory leak issue for org.apache.hadoop.hive.conf.HiveConf while using Flink into Hudi [hudi]

2023-11-08 Thread via GitHub
xmubeta opened a new issue, #10023: URL: https://github.com/apache/hudi/issues/10023 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at

Re: [PR] [HUDI-6872] Simplify Out Of Box Schema Evolution Functionality [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1802979926 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * e32b58f7ce1880568566be0c8a6940ae2f3a1016 UNKNOWN * 58d6194db3965823e985b646738d9d2399e4a5d8 Azure:

Re: [PR] [HUDI-6872] Simplify Out Of Box Schema Evolution Functionality [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1802973496 ## CI report: * 097ef6176650413eef2a4c3581ca6e48ea43788f UNKNOWN * e32b58f7ce1880568566be0c8a6940ae2f3a1016 UNKNOWN * 58d6194db3965823e985b646738d9d2399e4a5d8 Azure:

Re: [PR] [HUDI-7044] Skip reading records for delete blocks for positional merging [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10005: URL: https://github.com/apache/hudi/pull/10005#issuecomment-1802967731 ## CI report: * 83c12bf57803d832fbbfe2eed9cae30d987db175 Azure:

Re: [PR] [HUDI-6806] Support Spark 3.5.0 [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #9717: URL: https://github.com/apache/hudi/pull/9717#issuecomment-1802967250 ## CI report: * 9b8fdd2d1b69da528069e364790b53af1d6150af UNKNOWN * 94c46b0aaa5a205e767ed088ad631cc894922ea3 Azure:

Re: [I] UPSERTs are taking time [hudi]

2023-11-08 Thread via GitHub
soumilshah1995 commented on issue #9976: URL: https://github.com/apache/hudi/issues/9976#issuecomment-1802964892 can you use new RLI ?

Re: [I] [SUPPORT]insert_overwrite mode writing 2 times more duplicates [hudi]

2023-11-08 Thread via GitHub
soumilshah1995 commented on issue #9992: URL: https://github.com/apache/hudi/issues/9992#issuecomment-1802963658 here is sample works fine for me https://soumilshah1995.blogspot.com/2023/03/rfc-18-insert-overwrite-in-apache-hudi.html -- This is an automated message from the Apache

Re: [PR] [HUDI-7044] Skip reading records for delete blocks for positional merging [hudi]

2023-11-08 Thread via GitHub
hudi-bot commented on PR #10005: URL: https://github.com/apache/hudi/pull/10005#issuecomment-1802909950 ## CI report: * 579179dc1c43cf4fa02c3f023187ac9f8da06ffa Azure:

  1   2   3   >