Re: [PR] [HUDI-7128] DeleteProcedures support batch mode [hudi]

2023-11-21 Thread via GitHub
xuzifu666 closed pull request #10148: [HUDI-7128] DeleteProcedures support batch mode URL: https://github.com/apache/hudi/pull/10148 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [HUDI-7097] Fixing instantiation of Hms Uri with HiveSync tool [hudi]

2023-11-21 Thread via GitHub
nsivabalan commented on code in PR #10099: URL: https://github.com/apache/hudi/pull/10099#discussion_r1400158261 ## hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java: ## @@ -103,15 +103,29 @@ public class HiveSyncTool extends HoodieSyncTool

Re: [PR] [HUDI-7130] Adding support for configuring value serializer with JsonKakfaSource [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10149: URL: https://github.com/apache/hudi/pull/10149#issuecomment-1820431691 ## CI report: * e809a39b71dcfa3ddcfc6348b6740391b2a08dbd UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-6961] Fixing DefaultHoodieRecordPayload to honor deletion based on meta field as well as custome delete marker [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10150: URL: https://github.com/apache/hudi/pull/10150#issuecomment-1820629009 ## CI report: * ea3efa0db6b2a2e88508641d6ffb7eec9c33bf00 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-7041] Optimize the mem usage of partitionToFileGroupsMap during the cleaning [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10002: URL: https://github.com/apache/hudi/pull/10002#issuecomment-1820419758 ## CI report: * 03e698c23b11e2b1891b755ddae1e6431796680d Azure:

Re: [PR] [HUDI-7003] Add option to fallback to full table scan if files are deleted due to… [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #9941: URL: https://github.com/apache/hudi/pull/9941#issuecomment-1820430459 ## CI report: * 0a42984cc0d6d1e21b7e40b0fc08a8d6e902414c Azure:

Re: [PR] [HUDI-7129] Fix bug when upgrade from table version three using UpgradeOrDowngradeProcedure [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10147: URL: https://github.com/apache/hudi/pull/10147#issuecomment-1820547377 ## CI report: * 8b2189bea8fc0d58b17656bc429442f240530bc1 Azure:

[PR] [HUDI-6961] Fixing DefaultHoodieRecordPayload to honor deletion based on meta field as well as custome delete marker [hudi]

2023-11-21 Thread via GitHub
nsivabalan opened a new pull request, #10150: URL: https://github.com/apache/hudi/pull/10150 ### Change Logs [HUDI-6961] Fixing DefaultHoodieRecordPayload to honor deletion based on meta field as well as custom delete marker. ### Impact to be filled. ### Risk

Re: [PR] [HUDI-6961] Fixing DefaultHoodieRecordPayload to honor deletion based on meta field as well as custome delete marker [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10150: URL: https://github.com/apache/hudi/pull/10150#issuecomment-1820645211 ## CI report: * ea3efa0db6b2a2e88508641d6ffb7eec9c33bf00 Azure:

Re: [PR] [HUDI-7129] Fix bug when upgrade from table version three using UpgradeOrDowngradeProcedure [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10147: URL: https://github.com/apache/hudi/pull/10147#issuecomment-1820708498 ## CI report: * 8b2189bea8fc0d58b17656bc429442f240530bc1 Azure:

Re: [PR] [HUDI-7128] DeleteProcedures support batch mode [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10148: URL: https://github.com/apache/hudi/pull/10148#issuecomment-1820708583 ## CI report: * 785bae873c7e3a67c03b7516ba1bdf2cd18718c9 Azure:

Re: [PR] [HUDI-6734] Add back HUDI-5409: Avoid file index and use fs view cache in COW input format [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #9567: URL: https://github.com/apache/hudi/pull/9567#issuecomment-1820429742 ## CI report: * 4bdf55f7a59cf71b49b3b1d0ad2100390cfbe440 Azure:

Re: [PR] [HUDI-7083] Adding support for multiple tables with Prometheus Reporter [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10068: URL: https://github.com/apache/hudi/pull/10068#issuecomment-1820420024 ## CI report: * b91463465fe0eee81d69706909a877a8d4737556 Azure:

Re: [PR] [HUDI-7106] Fix sqs deletes, deltasync service close and error table default configs. [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10117: URL: https://github.com/apache/hudi/pull/10117#issuecomment-1820431193 ## CI report: * d34454306916251f8548db0e2729afbbf178e025 Azure:

Re: [PR] [HUDI-7115] Add in new options for the bigquery sync [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10125: URL: https://github.com/apache/hudi/pull/10125#issuecomment-1820431328 ## CI report: * d94d74a02df88f3ca32807c7f580900b268ca0d0 UNKNOWN * 2d743121c2c2fd4d228bc0db8b22598da592800a Azure:

Re: [PR] [HUDI-7084] Fixing schema retrieval for table w/ no commits [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10069: URL: https://github.com/apache/hudi/pull/10069#issuecomment-1820430882 ## CI report: * e266a298e0b7190b1e2de46619dcf8244f63e585 Azure:

Re: [PR] [HUDI-7097] Fixing instantiation of Hms Uri with HiveSync tool [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10099: URL: https://github.com/apache/hudi/pull/10099#issuecomment-1820431041 ## CI report: * 024770e006281bdea0ef90a8c4ba69b7d75d3fca Azure:

Re: [PR] [MINOR] Misc fixes in deltastreamer [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10067: URL: https://github.com/apache/hudi/pull/10067#issuecomment-1820430716 ## CI report: * eeb429c72ed8b46790e696a15637023a370784c9 UNKNOWN * e4a8a96631868cc5cd51f3a26c97750ac6d210c9 Azure:

Re: [PR] [HUDI-7083] Adding support for multiple tables with Prometheus Reporter [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10068: URL: https://github.com/apache/hudi/pull/10068#issuecomment-1820430795 ## CI report: * b91463465fe0eee81d69706909a877a8d4737556 Azure:

Re: [PR] [HUDI-7120] Performance improvements in deltastreamer executor code path [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10135: URL: https://github.com/apache/hudi/pull/10135#issuecomment-1820431435 ## CI report: * 2e26a7d1b87f4ca4e1f818612decfe0eb130a5fb Azure:

Re: [PR] [HUDI-7129] Fix bug when upgrade from table version three using UpgradeOrDowngradeProcedure [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10147: URL: https://github.com/apache/hudi/pull/10147#issuecomment-1820431600 ## CI report: * 8b2189bea8fc0d58b17656bc429442f240530bc1 Azure:

Re: [PR] [HUDI-7110] Add call procedure for show column stats information [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10120: URL: https://github.com/apache/hudi/pull/10120#issuecomment-1820504802 ## CI report: * 32ebcba726705152e2d539b0a90e998f2aa0b2ae Azure:

Re: [PR] [HUDI-7120] Performance improvements in deltastreamer executor code path [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10135: URL: https://github.com/apache/hudi/pull/10135#issuecomment-1820504957 ## CI report: * 4913158456e1dfaa1366ba7bd5029578f3bf4cef Azure:

Re: [PR] [HUDI-7083] Adding support for multiple tables with Prometheus Reporter [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10068: URL: https://github.com/apache/hudi/pull/10068#issuecomment-1820504529 ## CI report: * 5f8adafa3d8c7a8d3982c4d3dab4ecba174ef902 Azure:

Re: [PR] [HUDI-7003] Add option to fallback to full table scan if files are deleted due to… [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #9941: URL: https://github.com/apache/hudi/pull/9941#issuecomment-1820442384 ## CI report: * 0a42984cc0d6d1e21b7e40b0fc08a8d6e902414c Azure:

Re: [PR] [HUDI-7097] Fixing instantiation of Hms Uri with HiveSync tool [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10099: URL: https://github.com/apache/hudi/pull/10099#issuecomment-1820442950 ## CI report: * 024770e006281bdea0ef90a8c4ba69b7d75d3fca Azure:

Re: [PR] [MINOR] Misc fixes in deltastreamer [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10067: URL: https://github.com/apache/hudi/pull/10067#issuecomment-1820442718 ## CI report: * eeb429c72ed8b46790e696a15637023a370784c9 UNKNOWN * e4a8a96631868cc5cd51f3a26c97750ac6d210c9 Azure:

Re: [PR] [HUDI-7084] Fixing schema retrieval for table w/ no commits [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10069: URL: https://github.com/apache/hudi/pull/10069#issuecomment-1820442806 ## CI report: * e266a298e0b7190b1e2de46619dcf8244f63e585 Azure:

Re: [PR] [HUDI-6734] Add back HUDI-5409: Avoid file index and use fs view cache in COW input format [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #9567: URL: https://github.com/apache/hudi/pull/9567#issuecomment-1820441665 ## CI report: * 4bdf55f7a59cf71b49b3b1d0ad2100390cfbe440 Azure:

Re: [PR] [HUDI-7106] Fix sqs deletes, deltasync service close and error table default configs. [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10117: URL: https://github.com/apache/hudi/pull/10117#issuecomment-1820443112 ## CI report: * d34454306916251f8548db0e2729afbbf178e025 Azure:

Re: [PR] [HUDI-7129] Fix bug when upgrade from table version three using UpgradeOrDowngradeProcedure [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10147: URL: https://github.com/apache/hudi/pull/10147#issuecomment-1820443539 ## CI report: * 8b2189bea8fc0d58b17656bc429442f240530bc1 Azure:

Re: [PR] [HUDI-7130] Adding support for configuring value serializer with JsonKakfaSource [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10149: URL: https://github.com/apache/hudi/pull/10149#issuecomment-1820443632 ## CI report: * e809a39b71dcfa3ddcfc6348b6740391b2a08dbd Azure:

Re: [PR] [HUDI-7115] Add in new options for the bigquery sync [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10125: URL: https://github.com/apache/hudi/pull/10125#issuecomment-1820443298 ## CI report: * d94d74a02df88f3ca32807c7f580900b268ca0d0 UNKNOWN * 2d743121c2c2fd4d228bc0db8b22598da592800a Azure:

Re: [PR] [HUDI-7120] Performance improvements in deltastreamer executor code path [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10135: URL: https://github.com/apache/hudi/pull/10135#issuecomment-1820443377 ## CI report: * 2e26a7d1b87f4ca4e1f818612decfe0eb130a5fb Azure:

Re: [PR] [HUDI-7110] Add call procedure for show column stats information [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10120: URL: https://github.com/apache/hudi/pull/10120#issuecomment-1820443211 ## CI report: * 32ebcba726705152e2d539b0a90e998f2aa0b2ae Azure:

Re: [PR] [HUDI-7112] Reuse existing timeline server and performance improvements [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10122: URL: https://github.com/apache/hudi/pull/10122#issuecomment-1820533361 ## CI report: * c78d9570c87afce7a27725b37a3ebb77913199b7 Azure:

Re: [PR] [HUDI-7105] support filesystem view configuable to avoid clean oom [hudi]

2023-11-21 Thread via GitHub
danny0405 commented on PR #10116: URL: https://github.com/apache/hudi/pull/10116#issuecomment-1820582897 > We provide a config to allow using secondary table view first is also a optional way So only the clean task uses the local fs view ? I didn't see that change in Flink. --

Re: [PR] [HUDI-6961] Fixing DefaultHoodieRecordPayload to honor deletion based on meta field as well as custome delete marker [hudi]

2023-11-21 Thread via GitHub
nsivabalan commented on PR #10150: URL: https://github.com/apache/hudi/pull/10150#issuecomment-1820587093 this is a re-attempt of https://github.com/apache/hudi/pull/9892 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [HUDI-6961] Fix deletes with custom delete field in DefaultHoodieRecordPayload [hudi]

2023-11-21 Thread via GitHub
nsivabalan commented on PR #9892: URL: https://github.com/apache/hudi/pull/9892#issuecomment-1820589141 attempting a diff approach here https://github.com/apache/hudi/pull/10150 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [HUDI-7129] Fix bug when upgrade from table version three using UpgradeOrDowngradeProcedure [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10147: URL: https://github.com/apache/hudi/pull/10147#issuecomment-1820519474 ## CI report: * 8b2189bea8fc0d58b17656bc429442f240530bc1 Azure:

Re: [PR] [HUDI-7112] Reuse existing timeline server and performance improvements [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10122: URL: https://github.com/apache/hudi/pull/10122#issuecomment-1820519250 ## CI report: * c78d9570c87afce7a27725b37a3ebb77913199b7 Azure:

Re: [I] [SUPPORT] Getting Error received while writing records for transaction 20231119141913105 in partition 0 (org.apache.hudi.connect.transaction.ConnectTransactionParticipant:238) java.lang.NullPo

2023-11-21 Thread via GitHub
seethb commented on issue #10142: URL: https://github.com/apache/hudi/issues/10142#issuecomment-1820568103 Hi I have tried AVRO converter JARs from conflunet https://www.confluent.io/hub/confluentinc/kafka-connect-avro-converter and copied to our plugin directory and restarted.

Re: [I] [SUPPORT][BUG][DATA_LOSS] Refresh table does not work in 0.13.1 [hudi]

2023-11-21 Thread via GitHub
VitoMakarevich commented on issue #10088: URL: https://github.com/apache/hudi/issues/10088#issuecomment-1820821968 I created a PR with fix https://github.com/apache/hudi/pull/10151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] [HUDI-7034] Refresh index fix - remove cached file slices within part… [hudi]

2023-11-21 Thread via GitHub
VitoMakarevich opened a new pull request, #10151: URL: https://github.com/apache/hudi/pull/10151 ### Change Logs This is a fix for the issue [10088](https://github.com/apache/hudi/issues/10088)/[Hudi-7034](https://issues.apache.org/jira/browse/HUDI-7034). The issue is reproducible

Re: [PR] [HUDI-7034] Refresh index fix - remove cached file slices within part… [hudi]

2023-11-21 Thread via GitHub
VitoMakarevich commented on PR #10151: URL: https://github.com/apache/hudi/pull/10151#issuecomment-1820824105 @alexeykudinkin please take a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [HUDI-7083] Adding support for multiple tables with Prometheus Reporter [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10068: URL: https://github.com/apache/hudi/pull/10068#issuecomment-1820997375 ## CI report: * 0a84e366f03d44c1cc4ba67f5d4d1d6d76632d0b Azure:

[PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]

2023-11-21 Thread via GitHub
codope opened a new pull request, #10153: URL: https://github.com/apache/hudi/pull/10153 ### Change Logs With files and functional index, it should be possible to read the table from spark-sql attached to an external catalog, say remote Hive Metastore, even when the partition

[jira] [Updated] (HUDI-7023) Support querying without syncing partition metadata to catalog

2023-11-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7023: - Labels: pull-request-available (was: ) > Support querying without syncing partition metadata to

[jira] [Updated] (HUDI-7023) Support querying without syncing partition metadata to catalog

2023-11-21 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7023: -- Status: In Progress (was: Open) > Support querying without syncing partition metadata to catalog >

Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10153: URL: https://github.com/apache/hudi/pull/10153#issuecomment-1821029238 ## CI report: * 46a4c3344c79fd9a61db78620e8c40e7d98bcd36 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-7034] Refresh index fix - remove cached file slices within part… [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10151: URL: https://github.com/apache/hudi/pull/10151#issuecomment-1820838236 ## CI report: * b124e2a54cd9b3fec6d19c7c131b93234cd8c68c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [I] [SUPPORT] HUDI MOR table type compaction failed post adding new field in the schema [hudi]

2023-11-21 Thread via GitHub
abhisheksahani91 commented on issue #10138: URL: https://github.com/apache/hudi/issues/10138#issuecomment-1820976149 @ad1happy2go Today also I tried from scratch. At first, I inserting the records and later I changed the schema to add new field and send the update with new column This

[jira] [Assigned] (HUDI-7023) Support querying without syncing partition metadata to catalog

2023-11-21 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit reassigned HUDI-7023: - Assignee: Sagar Sumit > Support querying without syncing partition metadata to catalog >

[jira] [Updated] (HUDI-7023) Support querying without syncing partition metadata to catalog

2023-11-21 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-7023: -- Status: Patch Available (was: In Progress) > Support querying without syncing partition metadata to

Re: [PR] [HUDI-7120] Performance improvements in deltastreamer executor code path [hudi]

2023-11-21 Thread via GitHub
codope commented on PR #10135: URL: https://github.com/apache/hudi/pull/10135#issuecomment-1821037286 @lokeshj1703 There is a test failure, can you take a look - https://github.com/apache/hudi/actions/runs/6940682251/job/18889779707?pr=10135#step:6:20607 ``` Test alter column

Re: [I] [SUPPORT] HUDI MOR table type compaction failed post adding new field in the schema [hudi]

2023-11-21 Thread via GitHub
ad1happy2go commented on issue #10138: URL: https://github.com/apache/hudi/issues/10138#issuecomment-1820968721 @abhisheksahani91 I somehow tried a lot to reproduce the issue in my local setup with 0.12.1 Hudi version but unable to reproduce. Can you try to reproduce once like below -

Re: [PR] [HUDI-7004] Add support of snapshotLoadQuerySplitter(interface) in s3/gcs sources [hudi]

2023-11-21 Thread via GitHub
harsh1231 closed pull request #9943: [HUDI-7004] Add support of snapshotLoadQuerySplitter(interface) in s3/gcs sources URL: https://github.com/apache/hudi/pull/9943 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[jira] [Updated] (HUDI-7034) Refresh view does not work(due to cache)

2023-11-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-7034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7034: - Labels: pull-request-available (was: ) > Refresh view does not work(due to cache) >

[PR] Add support of snapshotLoadQuerySplitter in s3/gcs sources [hudi]

2023-11-21 Thread via GitHub
harsh1231 opened a new pull request, #10152: URL: https://github.com/apache/hudi/pull/10152 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any

Re: [PR] [HUDI-7004] Add support of snapshotLoadQuerySplitter in s3/gcs sources [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10152: URL: https://github.com/apache/hudi/pull/10152#issuecomment-1820915194 ## CI report: * 9764bc6527e5e3e83ed08263484beb45c1796d47 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

Re: [PR] [HUDI-7004] Add support of snapshotLoadQuerySplitter in s3/gcs sources [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10152: URL: https://github.com/apache/hudi/pull/10152#issuecomment-1820928772 ## CI report: * 9764bc6527e5e3e83ed08263484beb45c1796d47 UNKNOWN * d0fe92994777e2067d654e2585c75c91616f8598 UNKNOWN Bot commands @hudi-bot supports the

Re: [PR] [MINOR] Misc fixes in deltastreamer [hudi]

2023-11-21 Thread via GitHub
codope merged PR #10067: URL: https://github.com/apache/hudi/pull/10067 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(hudi) branch master updated (0c4f3a3164c -> baffe1d35f0)

2023-11-21 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 0c4f3a3164c [HUDI-7127] Fixing set up and tear down in tests (#10146) add baffe1d35f0 [MINOR] Misc fixes in

Re: [PR] [HUDI-7034] Refresh index fix - remove cached file slices within part… [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10151: URL: https://github.com/apache/hudi/pull/10151#issuecomment-1820900692 ## CI report: * b124e2a54cd9b3fec6d19c7c131b93234cd8c68c Azure:

Re: [PR] [MINOR] support log index [hudi]

2023-11-21 Thread via GitHub
watermelon12138 commented on PR #10143: URL: https://github.com/apache/hudi/pull/10143#issuecomment-1820993628 > Can you wrap up a general design of the changes, so that we are more eaiser to reach concensus for the general direction. @danny0405 ok, I will summarize the overall

Re: [PR] [HUDI-7004] Add support of snapshotLoadQuerySplitter in s3/gcs sources [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10152: URL: https://github.com/apache/hudi/pull/10152#issuecomment-1820998113 ## CI report: * 9764bc6527e5e3e83ed08263484beb45c1796d47 UNKNOWN * d0fe92994777e2067d654e2585c75c91616f8598 Azure:

Re: [PR] [HUDI-7112] Reuse existing timeline server and performance improvements [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10122: URL: https://github.com/apache/hudi/pull/10122#issuecomment-1821683706 ## CI report: * 697114b6ec4f578123363a89a6846e352bc3a53e Azure:

Re: [PR] [HUDI-6961] Fixing DefaultHoodieRecordPayload to honor deletion based on meta field as well as custome delete marker [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10150: URL: https://github.com/apache/hudi/pull/10150#issuecomment-1821715566 ## CI report: * ea3efa0db6b2a2e88508641d6ffb7eec9c33bf00 Azure:

Re: [PR] [HUDI-7130] Adding support for configuring value serializer with JsonKakfaSource [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10149: URL: https://github.com/apache/hudi/pull/10149#issuecomment-1821620236 ## CI report: * e809a39b71dcfa3ddcfc6348b6740391b2a08dbd Azure:

Re: [PR] [HUDI-7129] Fix bug when upgrade from table version three using UpgradeOrDowngradeProcedure [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10147: URL: https://github.com/apache/hudi/pull/10147#issuecomment-1821620165 ## CI report: * 1dee5fb303eff272371c638d07d80806676fd5aa Azure:

Re: [PR] [HUDI-7115] Add in new options for the bigquery sync [hudi]

2023-11-21 Thread via GitHub
nsivabalan merged PR #10125: URL: https://github.com/apache/hudi/pull/10125 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(hudi) branch master updated: [HUDI-7115] Add in new options for the bigquery sync (#10125)

2023-11-21 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new a1afcdd989c [HUDI-7115] Add in new options for

Re: [PR] [HUDI-7120] Performance improvements in deltastreamer executor code path [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10135: URL: https://github.com/apache/hudi/pull/10135#issuecomment-1821703416 ## CI report: * 3d48bfc5c41a59a1114eb73a5ef9a7b7fda5eccf Azure:

Re: [PR] [HUDI-6961] Fixing DefaultHoodieRecordPayload to honor deletion based on meta field as well as custome delete marker [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10150: URL: https://github.com/apache/hudi/pull/10150#issuecomment-1821703634 ## CI report: * ea3efa0db6b2a2e88508641d6ffb7eec9c33bf00 Azure:

Re: [PR] [HUDI-7110] Add call procedure for show column stats information [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10120: URL: https://github.com/apache/hudi/pull/10120#issuecomment-1821786342 ## CI report: * 03451f7cd016ee9fb078f4d78f3b771e8719c233 Azure:

Re: [PR] [HUDI-7129] Fix bug when upgrade from table version three using UpgradeOrDowngradeProcedure [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10147: URL: https://github.com/apache/hudi/pull/10147#issuecomment-1821842256 ## CI report: * 994d062df78afd5062dec418cddff167daff42d8 Azure:

Re: [PR] [HUDI-7034] Refresh index fix - remove cached file slices within part… [hudi]

2023-11-21 Thread via GitHub
VitoMakarevich commented on PR #10151: URL: https://github.com/apache/hudi/pull/10151#issuecomment-1821859085 Added a test, at least `fileIndex`-related tests pass(I ran only them), also can verify that executing `.refresh` on existing Index does not refresh a list of files as reported,

Re: [PR] [HUDI-7102] Fix a bug for time travel queries on MOR tables [hudi]

2023-11-21 Thread via GitHub
linliu-code commented on code in PR #10102: URL: https://github.com/apache/hudi/pull/10102#discussion_r1401341463 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/BaseHoodieLogRecordReader.java: ## @@ -260,7 +260,7 @@ private void scanInternalV1(Option keySpecOpt)

Re: [PR] [HUDI-7102] Fix a bug for time travel queries on MOR tables [hudi]

2023-11-21 Thread via GitHub
linliu-code commented on code in PR #10102: URL: https://github.com/apache/hudi/pull/10102#discussion_r1401341463 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/BaseHoodieLogRecordReader.java: ## @@ -260,7 +260,7 @@ private void scanInternalV1(Option keySpecOpt)

Re: [PR] [HUDI-7102] Fix a bug for time travel queries on MOR tables [hudi]

2023-11-21 Thread via GitHub
linliu-code commented on code in PR #10102: URL: https://github.com/apache/hudi/pull/10102#discussion_r1401341463 ## hudi-common/src/main/java/org/apache/hudi/common/table/log/BaseHoodieLogRecordReader.java: ## @@ -260,7 +260,7 @@ private void scanInternalV1(Option keySpecOpt)

Re: [PR] [HUDI-7102] Fix a bug for time travel queries on MOR tables [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10102: URL: https://github.com/apache/hudi/pull/10102#issuecomment-1821895011 ## CI report: * c3ff2511a30564e5a5ff0cb407326ff6ef0584e3 Azure:

Re: [PR] [HUDI-7034] Refresh index fix - remove cached file slices within part… [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10151: URL: https://github.com/apache/hudi/pull/10151#issuecomment-1821895329 ## CI report: * b124e2a54cd9b3fec6d19c7c131b93234cd8c68c Azure:

Re: [PR] [HUDI-7097] Fixing instantiation of Hms Uri with HiveSync tool [hudi]

2023-11-21 Thread via GitHub
the-other-tim-brown commented on code in PR #10099: URL: https://github.com/apache/hudi/pull/10099#discussion_r1400872147 ## hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java: ## @@ -103,15 +103,29 @@ public class HiveSyncTool extends HoodieSyncTool

[I] Fail to add default partition [hudi]

2023-11-21 Thread via GitHub
njalan opened a new issue, #10154: URL: https://github.com/apache/hudi/issues/10154 I got below error message: Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed in executing SQL ALTER TABLE `ods_xxx`.`xx` ADD IF NOT EXISTS PARTITION

Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10153: URL: https://github.com/apache/hudi/pull/10153#issuecomment-1821106645 ## CI report: * 46a4c3344c79fd9a61db78620e8c40e7d98bcd36 Azure:

Re: [PR] [HUDI-7023] Support querying without syncing partition metadata to catalog [hudi]

2023-11-21 Thread via GitHub
codope commented on code in PR #10153: URL: https://github.com/apache/hudi/pull/10153#discussion_r1400887742 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala: ## @@ -140,6 +146,68 @@ class TestFunctionalIndex

Re: [PR] [HUDI-7083] Adding support for multiple tables with Prometheus Reporter [hudi]

2023-11-21 Thread via GitHub
codope commented on PR #10068: URL: https://github.com/apache/hudi/pull/10068#issuecomment-1821081520 Unrelated to the PR, `testMultiWriterWithAsyncTableServicesWithConflict` failed which is flaky. Landing this PR. -- This is an automated message from the Apache Git Service. To respond

Re: [PR] [HUDI-7083] Adding support for multiple tables with Prometheus Reporter [hudi]

2023-11-21 Thread via GitHub
codope merged PR #10068: URL: https://github.com/apache/hudi/pull/10068 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(hudi) branch master updated: [HUDI-7083] Adding support for multiple tables with Prometheus Reporter (#10068)

2023-11-21 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 9e2500c8790 [HUDI-7083] Adding support for

Re: [PR] [HUDI-7130] Adding support for configuring value serializer with JsonKakfaSource [hudi]

2023-11-21 Thread via GitHub
vamshigv commented on code in PR #10149: URL: https://github.com/apache/hudi/pull/10149#discussion_r1400839419 ## hudi-utilities/pom.xml: ## @@ -368,6 +368,17 @@ kafka-schema-registry-client ${confluent.version} + Review Comment: Do we need to

Re: [I] [SUPPORT] HUDI MOR table type compaction failed post adding new field in the schema [hudi]

2023-11-21 Thread via GitHub
abhisheksahani91 commented on issue #10138: URL: https://github.com/apache/hudi/issues/10138#issuecomment-1821325376 @ad1happy2go Schema evolution is working now. I Did not change anything further and added the same properties you mentioned. The only issue is connection was

(hudi) branch master updated (600fd4dbdf5 -> b9818777f3e)

2023-11-21 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 600fd4dbdf5 [HUDI-6734] Add back HUDI-5409: Avoid file index and use fs view cache in COW input format (#9567)

Re: [PR] [HUDI-7003] Add option to fallback to full table scan if files are deleted due to… [hudi]

2023-11-21 Thread via GitHub
codope merged PR #9941: URL: https://github.com/apache/hudi/pull/9941 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [HUDI-6734] Add back HUDI-5409: Avoid file index and use fs view cache in COW input format [hudi]

2023-11-21 Thread via GitHub
codope merged PR #9567: URL: https://github.com/apache/hudi/pull/9567 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(hudi) branch master updated (9e2500c8790 -> 600fd4dbdf5)

2023-11-21 Thread codope
This is an automated email from the ASF dual-hosted git repository. codope pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 9e2500c8790 [HUDI-7083] Adding support for multiple tables with Prometheus Reporter (#10068) add 600fd4dbdf5

[jira] [Closed] (HUDI-6734) Add back HUDI-5409 in Hudi 0.12.x branch

2023-11-21 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-6734. - Resolution: Fixed > Add back HUDI-5409 in Hudi 0.12.x branch > >

[jira] [Updated] (HUDI-6734) Add back HUDI-5409 in Hudi 0.12.x branch

2023-11-21 Thread Sagar Sumit (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-6734: -- Fix Version/s: 0.14.1 > Add back HUDI-5409 in Hudi 0.12.x branch >

Re: [PR] [HUDI-7003] Add option to fallback to full table scan if files are deleted due to… [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #9941: URL: https://github.com/apache/hudi/pull/9941#issuecomment-1821260999 ## CI report: * 11c14c8e60bd8fd59e904f7c71af58b4073694a0 Azure:

Re: [PR] [HUDI-7112] Reuse existing timeline server and performance improvements [hudi]

2023-11-21 Thread via GitHub
nsivabalan commented on code in PR #10122: URL: https://github.com/apache/hudi/pull/10122#discussion_r1400959986 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/embedded/EmbeddedTimelineService.java: ## @@ -146,19 +214,65 @@ public FileSystemViewManager

Re: [PR] [HUDI-7084] Fixing schema retrieval for table w/ no commits [hudi]

2023-11-21 Thread via GitHub
hudi-bot commented on PR #10069: URL: https://github.com/apache/hudi/pull/10069#issuecomment-1821414245 ## CI report: * 14e044c14745918c7681554262be76f5427c82c5 Azure:

  1   2   >