[GitHub] [hudi] hudi-bot commented on pull request #7694: [HUDI-5572] Flink write need to skip check the compatibility of Schem…

hudi-bot commented on PR #7694: URL: https://github.com/apache/hudi/pull/7694#issuecomment-1397107642 ## CI report: * 97fdc558722b8d5152f9e21112045adb73eca9fe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443

[GitHub] [hudi] leobiscassi commented on issue #7533: [SUPPORT] Recreate deleted metadata table

leobiscassi commented on issue #7533: URL: https://github.com/apache/hudi/issues/7533#issuecomment-1397068362 Just to let you know @nsivabalan, your recommendation didn't work. I disabled (`hoodie.metadata.enable=false`) for 2 days and enabled after that (`hoodie.metadata.enable=true`) and

[GitHub] [hudi] koochiswathiTR opened a new issue, #7708: Parquet files are in small size

koochiswathiTR opened a new issue, #7708: URL: https://github.com/apache/hudi/issues/7708 Hi, We ingest data using spark streaming to hudi buckets. - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations a

[GitHub] [hudi] hudi-bot commented on pull request #7159: [HUDI-5173]Skip if there is only one file in clusteringGroup

hudi-bot commented on PR #7159: URL: https://github.com/apache/hudi/pull/7159#issuecomment-1396971957 ## CI report: * 15ecd91180d32c7fa1905c11408f4bc23347e682 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #7704: [HUDI-5583] multiple writers dead lock while cleaning

hudi-bot commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-1396889875 ## CI report: * 0ee4dc0391d7e1d4b0e155ceabf2a16f4de160bb UNKNOWN * 30a4839c829e13625dea78f93f39d5f6f80374d6 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-

[GitHub] [hudi] hudi-bot commented on pull request #7704: [HUDI-5583] multiple writers dead lock while cleaning

hudi-bot commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-1396879356 ## CI report: * 0ee4dc0391d7e1d4b0e155ceabf2a16f4de160bb UNKNOWN * 5fb5b2dcf3539cf39f2b4b4064047b22625a33d1 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-

[GitHub] [hudi] hudi-bot commented on pull request #7703: [HUDI-1575][DO NOT MERGE] Testing early conflict detection with feature flag enabled by default

hudi-bot commented on PR #7703: URL: https://github.com/apache/hudi/pull/7703#issuecomment-1396879248 ## CI report: * 60e2e430cd0915cc2270cad6277d6327c9f47a15 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1442

[GitHub] [hudi] SteNicholas commented on a diff in pull request #7159: [HUDI-5173]Skip if there is only one file in clusteringGroup

SteNicholas commented on code in PR #7159: URL: https://github.com/apache/hudi/pull/7159#discussion_r1081174963 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/clustering/plan/strategy/FlinkSizeBasedClusteringPlanStrategy.java: ## @@ -70,9 +70,11 @@ protect

[GitHub] [hudi] SteNicholas commented on pull request #7159: [HUDI-5173]Skip if there is only one file in clusteringGroup

SteNicholas commented on PR #7159: URL: https://github.com/apache/hudi/pull/7159#issuecomment-1396859233 @voonhous, when there are many partitions with files falling below the file size threshold, there is no much unnecessary writes for async incremental clustering which could config the fi

[GitHub] [hudi] voonhous commented on pull request #7159: [HUDI-5173]Skip if there is only one file in clusteringGroup

voonhous commented on PR #7159: URL: https://github.com/apache/hudi/pull/7159#issuecomment-1396810456 Hmmm, for flink jobs that are running under append-only mode, partitions that are not written to may get clustered even though there are no data being written to it. If there are ma

[GitHub] [hudi] SteNicholas commented on pull request #7159: [HUDI-5173]Skip if there is only one file in clusteringGroup

SteNicholas commented on PR #7159: URL: https://github.com/apache/hudi/pull/7159#issuecomment-1396804052 @danny0405, @zhuanshenbsj1, IMO, the benefit of this pull request is not great for incremental clustering, meanwhile there is no benefit for full Clustering. Therefore, this change of sk

[GitHub] [hudi] fengjian428 commented on issue #7654: [SUPPORT] Starvation on Hudi Java Client in OCC mode

fengjian428 commented on issue #7654: URL: https://github.com/apache/hudi/issues/7654#issuecomment-1396788939 @hemanth-gowda-12 try to add this line .withFileSystemViewConfig(FileSystemViewStorageConfig.newBuilder().withStorageType(FileSystemViewStorageType.MEMORY).build())

[GitHub] [hudi] hudi-bot commented on pull request #7705: Consistent Hashing CI Test

hudi-bot commented on PR #7705: URL: https://github.com/apache/hudi/pull/7705#issuecomment-1396782763 ## CI report: * 595996dcdb2ff01bc00c6a715a432cd2cfa4590f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=144

[GitHub] [hudi] hudi-bot commented on pull request #7705: Consistent Hashing CI Test

hudi-bot commented on PR #7705: URL: https://github.com/apache/hudi/pull/7705#issuecomment-1396773081 ## CI report: * 595996dcdb2ff01bc00c6a715a432cd2cfa4590f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=144

[GitHub] [hudi] hudi-bot commented on pull request #7704: [HUDI-5583] multiple writers dead lock while cleaning

hudi-bot commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-1396773027 ## CI report: * 0ee4dc0391d7e1d4b0e155ceabf2a16f4de160bb UNKNOWN * 5fb5b2dcf3539cf39f2b4b4064047b22625a33d1 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-

[GitHub] [hudi] hudi-bot commented on pull request #7512: [HUDI-5417] support to read avro from non-legacy map/list in parquet log

hudi-bot commented on PR #7512: URL: https://github.com/apache/hudi/pull/7512#issuecomment-1396772162 ## CI report: * 1e8f894b80c3e6081426b2f3e0b527dba914efd3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1389

[GitHub] [hudi] hudi-bot commented on pull request #7705: Consistent Hashing CI Test

hudi-bot commented on PR #7705: URL: https://github.com/apache/hudi/pull/7705#issuecomment-1396763184 ## CI report: * 595996dcdb2ff01bc00c6a715a432cd2cfa4590f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1444

[GitHub] [hudi] hudi-bot commented on pull request #7704: [HUDI-5583] multiple writers dead lock while cleaning

hudi-bot commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-1396763089 ## CI report: * 0ee4dc0391d7e1d4b0e155ceabf2a16f4de160bb UNKNOWN * 36ba0ee2ecf5818604bfc570bf58f36538d7febc Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-

[GitHub] [hudi] hudi-bot commented on pull request #7680: [HUDI-5548] spark sql show|update hudi's table properties

hudi-bot commented on PR #7680: URL: https://github.com/apache/hudi/pull/7680#issuecomment-1396762872 ## CI report: * e7b709b7ae40097094d7ff21177dfbc3dcead4bd Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=144

[GitHub] [hudi] hudi-bot commented on pull request #7512: [HUDI-5417] support to read avro from non-legacy map/list in parquet log

hudi-bot commented on PR #7512: URL: https://github.com/apache/hudi/pull/7512#issuecomment-1396762169 ## CI report: * 1e8f894b80c3e6081426b2f3e0b527dba914efd3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1389

[GitHub] [hudi] hudi-bot commented on pull request #7490: [HUDI-5407][HUDI-5408] Fixing rollback in MDT to be eager

hudi-bot commented on PR #7490: URL: https://github.com/apache/hudi/pull/7490#issuecomment-1396762052 ## CI report: * 22358346b08efffa5b184b9171d47356f43683d6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1442

[GitHub] [hudi] hudi-bot commented on pull request #6133: [HUDI-1575] Early Conflict Detection For Multi-writer

hudi-bot commented on PR #6133: URL: https://github.com/apache/hudi/pull/6133#issuecomment-1396759475 ## CI report: * dbe3db845908d261baa5a1aa71d19e0db55816de UNKNOWN * 678cce4a9748cb54a90a559384a0cb0443082535 UNKNOWN * 6fc5bf1ce7921bf25acc3659565457264d8b9dc2 UNKNOWN * 0b

[GitHub] [hudi] hudi-bot commented on pull request #7680: [HUDI-5548] spark sql show|update hudi's table properties

hudi-bot commented on PR #7680: URL: https://github.com/apache/hudi/pull/7680#issuecomment-1396750986 ## CI report: * ae2fbad50151a2ceedd75629068c2d07426c44c9 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=144

[GitHub] [hudi] hudi-bot commented on pull request #7232: [HUDI-5235] clustering target size should larger than small file limit

hudi-bot commented on PR #7232: URL: https://github.com/apache/hudi/pull/7232#issuecomment-1396749385 ## CI report: * 08239e5b8d4d49da4b5b3d814233251f81b3d0b0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1439

[GitHub] [hudi] lucabem commented on issue #7487: [SUPPORT] S3 Buckets reached quota limit when reading from hudi tables

lucabem commented on issue #7487: URL: https://github.com/apache/hudi/issues/7487#issuecomment-1396742280 Hi @yihua. Im facing the same error on EMR on EKS using Hudi 0.12.1. Im having slow download S3 file on first stages (check if is empty stage), downloading 300MB parquet on 1 hou

[GitHub] [hudi] mahesh2247 closed issue #7688: [SUPPORT] Trying to write a glue job script for reflecting CDC delete (Data Pipelining Kinesis Streams to create Apache Hudi Table from AWS Glue Job) . w

mahesh2247 closed issue #7688: [SUPPORT] Trying to write a glue job script for reflecting CDC delete (Data Pipelining Kinesis Streams to create Apache Hudi Table from AWS Glue Job) . while Insert and update are working fine. Kindly help URL: https://github.com/apache/hudi/issues/7688 -- This

[GitHub] [hudi] mahesh2247 commented on issue #7688: [SUPPORT] Trying to write a glue job script for reflecting CDC delete (Data Pipelining Kinesis Streams to create Apache Hudi Table from AWS Glue Jo

mahesh2247 commented on issue #7688: URL: https://github.com/apache/hudi/issues/7688#issuecomment-1396739453 This issue is solved. Just add the config for delete and should be fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] hemanth-gowda-12 commented on issue #7654: [SUPPORT] Starvation on Hudi Java Client in OCC mode

hemanth-gowda-12 commented on issue #7654: URL: https://github.com/apache/hudi/issues/7654#issuecomment-1396728910 @fengjian428 , sure.. Please feel free to include the test code. Thanks for quick fix. I tried it out, did not come across deadlocks anymore with FS based lock provider. Ho

[GitHub] [hudi] wzx140 commented on a diff in pull request #7512: [HUDI-5417] support to read avro from non-legacy map/list in parquet log

wzx140 commented on code in PR #7512: URL: https://github.com/apache/hudi/pull/7512#discussion_r1081015530 ## hudi-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

[GitHub] [hudi] wzx140 commented on a diff in pull request #7512: [HUDI-5417] support to read avro from non-legacy map/list in parquet log

wzx140 commented on code in PR #7512: URL: https://github.com/apache/hudi/pull/7512#discussion_r1081015158 ## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSourceWithParquetLog.scala: ## @@ -0,0 +1,143 @@ +/* + * Licensed to the Apache Sof

[GitHub] [hudi] wzx140 commented on a diff in pull request #7512: [HUDI-5417] support to read avro from non-legacy map/list in parquet log

wzx140 commented on code in PR #7512: URL: https://github.com/apache/hudi/pull/7512#discussion_r1081014834 ## hudi-common/src/main/java/org/apache/parquet/avro/HoodieAvroParquetReaderBuilder.java: ## @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [hudi] wzx140 commented on a diff in pull request #7512: [HUDI-5417] support to read avro from non-legacy map/list in parquet log

wzx140 commented on code in PR #7512: URL: https://github.com/apache/hudi/pull/7512#discussion_r1081014295 ## hudi-common/src/main/java/org/apache/parquet/avro/HoodieAvroParquetReaderBuilder.java: ## @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [hudi] wzx140 commented on a diff in pull request #7512: [HUDI-5417] support to read avro from non-legacy map/list in parquet log

wzx140 commented on code in PR #7512: URL: https://github.com/apache/hudi/pull/7512#discussion_r1081013630 ## hudi-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

[GitHub] [hudi] zhuanshenbsj1 commented on pull request #7232: [HUDI-5235] clustering target size should larger than small file limit

zhuanshenbsj1 commented on PR #7232: URL: https://github.com/apache/hudi/pull/7232#issuecomment-1396672512 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [hudi] hudi-bot commented on pull request #7612: [HUDI-5336] Fixing log file pattern match to ignore extraneous files

hudi-bot commented on PR #7612: URL: https://github.com/apache/hudi/pull/7612#issuecomment-139322 ## CI report: * 1dc0a0732953fa0b470054c828981e226803e8aa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1437

[GitHub] [hudi] hudi-bot commented on pull request #7612: [HUDI-5336] Fixing log file pattern match to ignore extraneous files

hudi-bot commented on PR #7612: URL: https://github.com/apache/hudi/pull/7612#issuecomment-1396656076 ## CI report: * 1dc0a0732953fa0b470054c828981e226803e8aa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1437

[GitHub] [hudi] liaotian1005 commented on a diff in pull request #7633: Fix Deletes issued without any prior commits

liaotian1005 commented on code in PR #7633: URL: https://github.com/apache/hudi/pull/7633#discussion_r1080943658 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java: ## @@ -1637,8 +1637,6 @@ protected void setWriteSchemaForDeletes(H

[GitHub] [hudi] liaotian1005 commented on a diff in pull request #7633: Fix Deletes issued without any prior commits

liaotian1005 commented on code in PR #7633: URL: https://github.com/apache/hudi/pull/7633#discussion_r1080939641 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java: ## @@ -1637,8 +1637,6 @@ protected void setWriteSchemaForDeletes(H

[GitHub] [hudi] nsivabalan commented on a diff in pull request #7612: [HUDI-5336] Fixing log file pattern match to ignore extraneous files

nsivabalan commented on code in PR #7612: URL: https://github.com/apache/hudi/pull/7612#discussion_r1080921518 ## hudi-common/src/test/java/org/apache/hudi/common/functional/TestHoodieLogFormatAppendFailure.java: ## @@ -134,7 +134,7 @@ public void testFailedToGetAppendStreamFrom

[GitHub] [hudi] hudi-bot commented on pull request #6384: [HUDI-4613] Avoid the use of regex expressions when call hoodieFileGroup#addLogFile function

hudi-bot commented on PR #6384: URL: https://github.com/apache/hudi/pull/6384#issuecomment-1396567540 ## CI report: * a7ece0e42ac674d75b035220f129e5c0892dbf05 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1437

[GitHub] [hudi] hudi-bot commented on pull request #7707: [HUDI-5581] enabling scan V2 for log record reader to test CI

hudi-bot commented on PR #7707: URL: https://github.com/apache/hudi/pull/7707#issuecomment-1396562553 ## CI report: * f8ee15601abca026fc0b9f894a785e2a97291d88 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=144

[GitHub] [hudi] hudi-bot commented on pull request #7706: [HUDI-5585][flink]Fix flink creates and writes the table, the spark alter table reports an error

hudi-bot commented on PR #7706: URL: https://github.com/apache/hudi/pull/7706#issuecomment-1396562517 ## CI report: * 126951c4f2e2581ffbfb996df3d2ea325290f7f6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1444

[GitHub] [hudi] hudi-bot commented on pull request #7704: [HUDI-5583] multiple writers dead lock while cleaning

hudi-bot commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-1396562462 ## CI report: * 0ee4dc0391d7e1d4b0e155ceabf2a16f4de160bb UNKNOWN * 36ba0ee2ecf5818604bfc570bf58f36538d7febc Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-

[GitHub] [hudi] hudi-bot commented on pull request #7642: [HUDI-5534][Stacked on 6782] Optimizing Bloom Index lookup when using Bloom Filters from Metadata Table

hudi-bot commented on PR #7642: URL: https://github.com/apache/hudi/pull/7642#issuecomment-1396562097 ## CI report: * 6988dfce1a01dbb2e89bffabd0d2475134015756 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=144

[GitHub] [hudi] hudi-bot commented on pull request #6384: [HUDI-4613] Avoid the use of regex expressions when call hoodieFileGroup#addLogFile function

hudi-bot commented on PR #6384: URL: https://github.com/apache/hudi/pull/6384#issuecomment-1396560528 ## CI report: * a7ece0e42ac674d75b035220f129e5c0892dbf05 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1437

[GitHub] [hudi] hudi-bot commented on pull request #7707: [HUDI-5581] enabling scan V2 for log record reader to test CI

hudi-bot commented on PR #7707: URL: https://github.com/apache/hudi/pull/7707#issuecomment-1396556197 ## CI report: * f8ee15601abca026fc0b9f894a785e2a97291d88 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #7706: [HUDI-5585][flink]Fix flink creates and writes the table, the spark alter table reports an error

hudi-bot commented on PR #7706: URL: https://github.com/apache/hudi/pull/7706#issuecomment-1396556155 ## CI report: * 126951c4f2e2581ffbfb996df3d2ea325290f7f6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #7704: [HUDI-5583] multiple writers dead lock while cleaning

hudi-bot commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-1396556097 ## CI report: * 0f65ca4feef6eed0fabe5d88b96b1fb797834470 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=144

[GitHub] [hudi] hudi-bot commented on pull request #7642: [HUDI-5534][Stacked on 6782] Optimizing Bloom Index lookup when using Bloom Filters from Metadata Table

hudi-bot commented on PR #7642: URL: https://github.com/apache/hudi/pull/7642#issuecomment-1396555831 ## CI report: * 6988dfce1a01dbb2e89bffabd0d2475134015756 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=144

[GitHub] [hudi] hudi-bot commented on pull request #7704: [HUDI-5583] multiple writers dead lock while cleaning

hudi-bot commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-1396549252 ## CI report: * 0f65ca4feef6eed0fabe5d88b96b1fb797834470 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=144

[GitHub] [hudi] hudi-bot commented on pull request #7642: [HUDI-5534][Stacked on 6782] Optimizing Bloom Index lookup when using Bloom Filters from Metadata Table

hudi-bot commented on PR #7642: URL: https://github.com/apache/hudi/pull/7642#issuecomment-1396548870 ## CI report: * 6988dfce1a01dbb2e89bffabd0d2475134015756 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=144

[GitHub] [hudi] hudi-bot commented on pull request #7379: [HUDI-5323][Draft] Support virtual keys in Bloom Index and always write bloom filters to parquet files

hudi-bot commented on PR #7379: URL: https://github.com/apache/hudi/pull/7379#issuecomment-1396548297 ## CI report: * e007860f5b4151ca4e9a7109028f00dd4626a66f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1441

[GitHub] [hudi] yihua commented on a diff in pull request #7612: [HUDI-5336] Fixing log file pattern match to ignore extraneous files

yihua commented on code in PR #7612: URL: https://github.com/apache/hudi/pull/7612#discussion_r1080888468 ## hudi-common/src/test/java/org/apache/hudi/common/functional/TestHoodieLogFormatAppendFailure.java: ## @@ -134,7 +134,7 @@ public void testFailedToGetAppendStreamFromHDFSN

[GitHub] [hudi] nsivabalan closed pull request #7692: [HUDI-5581] enabling scan V2 for log record reader

nsivabalan closed pull request #7692: [HUDI-5581] enabling scan V2 for log record reader URL: https://github.com/apache/hudi/pull/7692 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[GitHub] [hudi] nsivabalan opened a new pull request, #7707: [HUDI-5581] enabling scan V2 for log record reader to test CI

nsivabalan opened a new pull request, #7707: URL: https://github.com/apache/hudi/pull/7707 ### Change Logs [HUDI-5581] enabling scan V2 for log record reader to test CI ### Impact _Describe any public API or user-facing feature change or any performance impact._ #

[GitHub] [hudi] nsivabalan commented on pull request #7692: [HUDI-5581] enabling scan V2 for log record reader

nsivabalan commented on PR #7692: URL: https://github.com/apache/hudi/pull/7692#issuecomment-1396531792 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions

alexeykudinkin commented on code in PR #7702: URL: https://github.com/apache/hudi/pull/7702#discussion_r1080866041 ## pom.xml: ## @@ -440,34 +441,13 @@ - - com.esotericsoftware:kryo-shaded - com.esoter

[GitHub] [hudi] waywtdcc opened a new pull request, #7706: [HUDI-5585][flink]Fix flink creates and writes the table, the spark alter table reports an error

waywtdcc opened a new pull request, #7706: URL: https://github.com/apache/hudi/pull/7706 ### Change Logs Fix flink creates and writes the table, the spark alter table reports an error ### Impact Fix flink creates and writes the table, the spark alter table report

[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions

alexeykudinkin commented on code in PR #7702: URL: https://github.com/apache/hudi/pull/7702#discussion_r1080862515 ## hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/HoodieSparkKryoRegistrar.scala: ## @@ -42,22 +43,31 @@ import org.apache.spark.serializer.KryoRegis

[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #7680: [HUDI-5548] spark sql show|update hudi's table properties

xiarixiaoyao commented on code in PR #7680: URL: https://github.com/apache/hudi/pull/7680#discussion_r1080858370 ## hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterTableCommand.scala: ## @@ -344,5 +359,21 @@ object AlterTableC

[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #7680: [HUDI-5548] spark sql show|update hudi's table properties

xiarixiaoyao commented on code in PR #7680: URL: https://github.com/apache/hudi/pull/7680#discussion_r1080858251 ## hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterTableCommand.scala: ## @@ -344,5 +359,21 @@ object AlterTableC

[GitHub] [hudi] hudi-bot commented on pull request #7705: Consistent Hashing CI Test

hudi-bot commented on PR #7705: URL: https://github.com/apache/hudi/pull/7705#issuecomment-1396493252 ## CI report: * 595996dcdb2ff01bc00c6a715a432cd2cfa4590f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1444

[GitHub] [hudi] hudi-bot commented on pull request #7704: [HUDI-5583] multiple writers dead lock while cleaning

hudi-bot commented on PR #7704: URL: https://github.com/apache/hudi/pull/7704#issuecomment-1396493214 ## CI report: * 0f65ca4feef6eed0fabe5d88b96b1fb797834470 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1443

[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions

hudi-bot commented on PR #7702: URL: https://github.com/apache/hudi/pull/7702#issuecomment-1396493142 ## CI report: * 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN * 27d54ea8f8f395f2044009d51ba8e969c74c2a4d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-

[GitHub] [hudi] hudi-bot commented on pull request #7677: [HUDI-5559] Support CDC for flink bounded source

hudi-bot commented on PR #7677: URL: https://github.com/apache/hudi/pull/7677#issuecomment-1396492975 ## CI report: * 19b72be84fa15d4dbd9f55c5b79b78dc4470159d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1438

[GitHub] [hudi] hudi-bot commented on pull request #7626: [HUDI-5516] Reduce memory footprint on workload with thousand active partitions

hudi-bot commented on PR #7626: URL: https://github.com/apache/hudi/pull/7626#issuecomment-1396492779 ## CI report: * 784dd7c8e7f8d6b7013071df04bfb57121b1d6c9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1438

[GitHub] [hudi] hudi-bot commented on pull request #7607: [HUDI-5499] Fixing Spark SQL configs not being properly propagated for CTAS and other commands

hudi-bot commented on PR #7607: URL: https://github.com/apache/hudi/pull/7607#issuecomment-1396492695 ## CI report: * 32033e4a4ed91005a237aa88afa2c6adcb51169f UNKNOWN * 05cbda8ddcca0944c7965bd7c32448e29f97 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #6815: [HUDI-4937][Stacked on 7702] Fix `HoodieTable` injecting non-reusable `HoodieBackedTableMetadata` aggressively flushing MT readers

hudi-bot commented on PR #6815: URL: https://github.com/apache/hudi/pull/6815#issuecomment-1396491598 ## CI report: * 13fb78850890b96b86b66d7df060feb11950ec0c UNKNOWN * 8d658d4564ef7e0aacd856657643fd722d4893ea Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-

[GitHub] [hudi] hudi-bot commented on pull request #7705: Consistent Hashing CI Test

hudi-bot commented on PR #7705: URL: https://github.com/apache/hudi/pull/7705#issuecomment-1396486555 ## CI report: * 595996dcdb2ff01bc00c6a715a432cd2cfa4590f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions

hudi-bot commented on PR #7702: URL: https://github.com/apache/hudi/pull/7702#issuecomment-1396486493 ## CI report: * 384a9774018272e13b967817b0e48b1596a23dcc UNKNOWN * 11cceab4fc10a46bb7d169079262b1e4c1ea558e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-

[GitHub] [hudi] hudi-bot commented on pull request #7677: [HUDI-5559] Support CDC for flink bounded source

hudi-bot commented on PR #7677: URL: https://github.com/apache/hudi/pull/7677#issuecomment-1396486353 ## CI report: * 19b72be84fa15d4dbd9f55c5b79b78dc4470159d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1438

[GitHub] [hudi] hudi-bot commented on pull request #7626: [HUDI-5516] Reduce memory footprint on workload with thousand active partitions

hudi-bot commented on PR #7626: URL: https://github.com/apache/hudi/pull/7626#issuecomment-1396486170 ## CI report: * 784dd7c8e7f8d6b7013071df04bfb57121b1d6c9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1438

[GitHub] [hudi] hudi-bot commented on pull request #7607: [HUDI-5499] Fixing Spark SQL configs not being properly propagated for CTAS and other commands

hudi-bot commented on PR #7607: URL: https://github.com/apache/hudi/pull/7607#issuecomment-1396486053 ## CI report: * 32033e4a4ed91005a237aa88afa2c6adcb51169f UNKNOWN * 05cbda8ddcca0944c7965bd7c32448e29f97 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2

[GitHub] [hudi] hudi-bot commented on pull request #6815: [HUDI-4937][Stacked on 7702] Fix `HoodieTable` injecting non-reusable `HoodieBackedTableMetadata` aggressively flushing MT readers

hudi-bot commented on PR #6815: URL: https://github.com/apache/hudi/pull/6815#issuecomment-1396485170 ## CI report: * 13fb78850890b96b86b66d7df060feb11950ec0c UNKNOWN * 987bcb8f3498e8b43fdef04683c7f4fadbb968fe Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-

[GitHub] [hudi] fengjian428 commented on issue #7654: [SUPPORT] Starvation on Hudi Java Client in OCC mode

fengjian428 commented on issue #7654: URL: https://github.com/apache/hudi/issues/7654#issuecomment-1396481587 > Cool, do we need a patch for this issue, can you fire a JIRA issue then @fengjian428? https://issues.apache.org/jira/browse/HUDI-5583 -- This is an automated message from

[GitHub] [hudi] hudi-bot commented on pull request #7700: [HUDI-5575] Support any record key generation along w/ any partition path generation for row writer

hudi-bot commented on PR #7700: URL: https://github.com/apache/hudi/pull/7700#issuecomment-1396480473 ## CI report: * 5c759d92fd68bbfdd716bdeb4bb18c1e4a4cd415 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1441

[GitHub] [hudi] LinMingQiang commented on a diff in pull request #7694: [HUDI-5572] Flink write need to skip check the compatibility of Schem…

LinMingQiang commented on code in PR #7694: URL: https://github.com/apache/hudi/pull/7694#discussion_r1080834901 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/HoodieFlinkTable.java: ## @@ -114,4 +114,11 @@ private static void setLatestInternalSchema(Hoodi

[GitHub] [hudi] LinMingQiang commented on a diff in pull request #7694: [HUDI-5572] Flink write need to skip check the compatibility of Schem…

LinMingQiang commented on code in PR #7694: URL: https://github.com/apache/hudi/pull/7694#discussion_r1080833821 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/HoodieFlinkTable.java: ## @@ -114,4 +114,11 @@ private static void setLatestInternalSchema(Hoodi

[GitHub] [hudi] danny0405 commented on a diff in pull request #7694: [HUDI-5572] Flink write need to skip check the compatibility of Schem…

danny0405 commented on code in PR #7694: URL: https://github.com/apache/hudi/pull/7694#discussion_r1080830408 ## hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/HoodieFlinkTable.java: ## @@ -114,4 +114,11 @@ private static void setLatestInternalSchema(HoodieWr

[GitHub] [hudi] trushev commented on pull request #7626: [HUDI-5516] Reduce memory footprint on workload with thousand active partitions

trushev commented on PR #7626: URL: https://github.com/apache/hudi/pull/7626#issuecomment-1396468460 @TengHuo I tried the following workload with MOR table, 2000 partitions and compaction (checkpoint here triggers compaction) ```java public class TestPartitionsWorkloadWithCompactio

[GitHub] [hudi] danny0405 commented on issue #7600: Hoodie clean is not deleting old files for MOR table

danny0405 commented on issue #7600: URL: https://github.com/apache/hudi/issues/7600#issuecomment-1396467854 cc @nsivabalan, can you take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [hudi] danny0405 commented on issue #7689: [SUPPORT] PriorityBasedFileSystemView: Got error running preferred function. Trying secondary

danny0405 commented on issue #7689: URL: https://github.com/apache/hudi/issues/7689#issuecomment-1396466926 Okey, let's keep this issue open and see if we have some space for the stability improvement for fs view. -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] danny0405 commented on issue #7683: [SUPPORT] Querying data using Trino only returns records of the latest commit, not all records.

danny0405 commented on issue #7683: URL: https://github.com/apache/hudi/issues/7683#issuecomment-1396466220 cc @codope, can you take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[GitHub] [hudi] danny0405 merged pull request #7696: [MINOR] add missing link and fix typo

danny0405 merged PR #7696: URL: https://github.com/apache/hudi/pull/7696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache

[GitHub] [hudi] danny0405 merged pull request #7698: [HUDI-5577] Validate option catalog.path in dfs mode

danny0405 merged PR #7698: URL: https://github.com/apache/hudi/pull/7698 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache

[GitHub] [hudi] danny0405 commented on a diff in pull request #7633: Fix Deletes issued without any prior commits

danny0405 commented on code in PR #7633: URL: https://github.com/apache/hudi/pull/7633#discussion_r1080824810 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java: ## @@ -1637,8 +1637,6 @@ protected void setWriteSchemaForDeletes(Hood

[GitHub] [hudi] danny0405 commented on a diff in pull request #7633: Fix Deletes issued without any prior commits

danny0405 commented on code in PR #7633: URL: https://github.com/apache/hudi/pull/7633#discussion_r1080824589 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java: ## @@ -1637,8 +1637,6 @@ protected void setWriteSchemaForDeletes(Hood

[GitHub] [hudi] danny0405 commented on issue #7688: [SUPPORT] Trying to write a glue job script for reflecting CDC delete (Data Pipelining Kinesis Streams to create Apache Hudi Table from AWS Glue Job

danny0405 commented on issue #7688: URL: https://github.com/apache/hudi/issues/7688#issuecomment-1396459234 Yeah, I guess kiness does not do any differential operation to the delete records, all the records are taken as INSERTs, that explain your test case. -- This is an automated message

[GitHub] [hudi] danny0405 commented on issue #7654: [SUPPORT] Starvation on Hudi Java Client in OCC mode

danny0405 commented on issue #7654: URL: https://github.com/apache/hudi/issues/7654#issuecomment-1396453511 Cool, do we need a patch for this issue, can you fire a JIRA issue then @fengjian428? -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hudi] mahesh2247 commented on issue #7688: [SUPPORT] Trying to write a glue job script for reflecting CDC delete (Data Pipelining Kinesis Streams to create Apache Hudi Table from AWS Glue Jo

mahesh2247 commented on issue #7688: URL: https://github.com/apache/hudi/issues/7688#issuecomment-1396453311 @danny0405 @dannyhchen @umehrot2 I don't think anybody has tried to insert, update and delete streaming data with hudi (kinesis , aws glue), i do not see much documentation as well a

[GitHub] [hudi] danny0405 commented on issue #7688: [SUPPORT] Trying to write a glue job script for reflecting CDC delete (Data Pipelining Kinesis Streams to create Apache Hudi Table from AWS Glue Job

danny0405 commented on issue #7688: URL: https://github.com/apache/hudi/issues/7688#issuecomment-1396451954 Not sure how kinesis handles the delete records, @umehrot2, can you clarify some details here? -- This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [hudi] mahesh2247 commented on issue #7688: [SUPPORT] Trying to write a glue job script for reflecting CDC delete (Data Pipelining Kinesis Streams to create Apache Hudi Table from AWS Glue Jo

mahesh2247 commented on issue #7688: URL: https://github.com/apache/hudi/issues/7688#issuecomment-1396451723 @danny0405 @dannyhchen I'm guessing i need to change something below ``` commonConfig = {'hoodie.datasource.write.hive_style_partitioning' : 'true','className' : 'org.apache

[GitHub] [hudi] lokeshj1703 opened a new pull request, #7705: Consistent Hashing CI Test

lokeshj1703 opened a new pull request, #7705: URL: https://github.com/apache/hudi/pull/7705 ### Change Logs Consistent Hashing CI Test ### Impact Na ### Risk level (write none, low medium or high below) NA ### Documentation Update _Describe an

[GitHub] [hudi] trushev commented on pull request #7626: [HUDI-5516] Reduce memory footprint on workload with thousand active partitions

trushev commented on PR #7626: URL: https://github.com/apache/hudi/pull/7626#issuecomment-1396449286 > Thanks for great effort @trushev, can we revert the changes/refactoring for `#performWriteOperation`, it is not related with this issue, and we can address it in another PR. > > We

[GitHub] [hudi] mahesh2247 commented on issue #7688: [SUPPORT] Trying to write a glue job script for reflecting CDC delete (Data Pipelining Kinesis Streams to create Apache Hudi Table from AWS Glue Jo

mahesh2247 commented on issue #7688: URL: https://github.com/apache/hudi/issues/7688#issuecomment-1396448951 Do I need to change anything in my code ? @danny0405 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] danny0405 commented on issue #7688: [SUPPORT] Trying to write a glue job script for reflecting CDC delete (Data Pipelining Kinesis Streams to create Apache Hudi Table from AWS Glue Job

danny0405 commented on issue #7688: URL: https://github.com/apache/hudi/issues/7688#issuecomment-1396448471 There is a metadata filed named `_hoodie_is_deleted`, if you mark that as true, Hudi would delete the records for you. -- This is an automated message from the Apache Git Service. T

[GitHub] [hudi] danny0405 commented on a diff in pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions

danny0405 commented on code in PR #7702: URL: https://github.com/apache/hudi/pull/7702#discussion_r1080809686 ## hudi-common/src/main/java/org/apache/hudi/common/util/HoodieCommonKryoRegistrar.java: ## @@ -81,7 +83,8 @@ public Class[] registerClasses() { HoodieRecordL

[GitHub] [hudi] mahesh2247 commented on issue #7688: [SUPPORT] Trying to write a glue job script for reflecting CDC delete (Data Pipelining Kinesis Streams to create Apache Hudi Table from AWS Glue Jo

mahesh2247 commented on issue #7688: URL: https://github.com/apache/hudi/issues/7688#issuecomment-1396445481 Hi @danny0405 and @umehrot2! Any suggestions on what configs needs to be changed to accomodate deletion ? Kindly help -- This is an automated message from the Apache Git Service

[GitHub] [hudi] BalaMahesh commented on issue #7657: [SUPPORT] Invalid number of file groups for partition:column_stats

BalaMahesh commented on issue #7657: URL: https://github.com/apache/hudi/issues/7657#issuecomment-1396439251 @nsivabalan - wrt to metadata indexing, we can enable this as async job within delta streamer job with below config right ? # ensure that both metadata and async indexing is e

[GitHub] [hudi] danny0405 commented on a diff in pull request #7702: [HUDI-5579] Fixing Kryo registration to be properly wired into Spark sessions

danny0405 commented on code in PR #7702: URL: https://github.com/apache/hudi/pull/7702#discussion_r1080806942 ## hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/HoodieSparkKryoRegistrar.scala: ## @@ -42,22 +43,31 @@ import org.apache.spark.serializer.KryoRegistrato

<    1   2   3   4   5   6   7   8   9   10   >