Re: [PR] [HUDI-6927] CDC file clean not work [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9841: URL: https://github.com/apache/hudi/pull/9841#issuecomment-1754406212 ## CI report: * 7cc75814a8d31fe885ce7538c963a40691d06a5c UNKNOWN * c2f7bfb45728ad79349d74c504caf804acd9e2c1 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1754405201 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * de2560dfe639845337ad6dd861e8cf980c2a3c3b Azure:

Re: [I] [SUPPORT] Process Fallout: Unable to upsert due to continues Failure in Compaction [hudi]

2023-10-09 Thread via GitHub
ketkidev commented on issue #9822: URL: https://github.com/apache/hudi/issues/9822#issuecomment-1754401081 @ad1happy2go We removed all inflight requests for compaction. Still unable to process further. -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] [SUPPORT]: Data loss with Concurrent operations on Hudi MOR [hudi]

2023-10-09 Thread via GitHub
ketkidev commented on issue #9674: URL: https://github.com/apache/hudi/issues/9674#issuecomment-1754394850 @ad1happy2go As suggested, tried apply `--hoodie-conf hoodie.cleaner.policy.failed.writes=LAZY` but still found data loss in few tables. -- This is an automated message from the

Re: [PR] [HUDI-6927] CDC file clean not work [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9841: URL: https://github.com/apache/hudi/pull/9841#issuecomment-1754354638 ## CI report: * 7cc75814a8d31fe885ce7538c963a40691d06a5c UNKNOWN * a2a9f0ee24ba017f4e0c6148d182df501cca3ad7 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1754354388 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * de2560dfe639845337ad6dd861e8cf980c2a3c3b Azure:

Re: [PR] [HUDI-6927] CDC file clean not work [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9841: URL: https://github.com/apache/hudi/pull/9841#issuecomment-1754342391 ## CI report: * 7cc75814a8d31fe885ce7538c963a40691d06a5c UNKNOWN * a2a9f0ee24ba017f4e0c6148d182df501cca3ad7 Azure:

Re: [PR] [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #8143: URL: https://github.com/apache/hudi/pull/8143#issuecomment-1754337249 ## CI report: * 0bd1545bc12e727c30a08e689c05fcc59c1a Azure:

Re: [PR] [HUDI-5220] fix hive snapshot query add non hoodie paths file status [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #7206: URL: https://github.com/apache/hudi/pull/7206#issuecomment-1754335306 ## CI report: * 5d7a1c4824c100a48c95e3d017822aa1062ad8cd Azure:

Re: [PR] [HUDI-6927] CDC file clean not work [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9841: URL: https://github.com/apache/hudi/pull/9841#issuecomment-1754306189 ## CI report: * 7cc75814a8d31fe885ce7538c963a40691d06a5c UNKNOWN * a2a9f0ee24ba017f4e0c6148d182df501cca3ad7 Azure:

Re: [PR] [HUDI-6926] Disable DROP_PARTITION_COLUMNS when upsert MOR table [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9840: URL: https://github.com/apache/hudi/pull/9840#issuecomment-1754306160 ## CI report: * e8bb2bf52d648aa10ad684f6b5ee35ee75d28fcc Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1754306053 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * de2560dfe639845337ad6dd861e8cf980c2a3c3b Azure:

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9581: URL: https://github.com/apache/hudi/pull/9581#issuecomment-1754305774 ## CI report: * 391fc1099c8b51221a0448d8ac9d74e3013c1cd1 Azure:

Re: [PR] [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #8143: URL: https://github.com/apache/hudi/pull/8143#issuecomment-1754304659 ## CI report: * 0bd1545bc12e727c30a08e689c05fcc59c1a Azure:

[jira] [Updated] (HUDI-6930) Eliminate the shouldLock flag for TimeGenerator

2023-10-09 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6930: - Description: In HUDI-1623, we introduced the TimeGenerator, but left a shouldLock flag on the interfaces

[jira] [Assigned] (HUDI-6930) Eliminate the shouldLock flag for TimeGenerator

2023-10-09 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen reassigned HUDI-6930: Assignee: Vinoth Chandar > Eliminate the shouldLock flag for TimeGenerator >

Re: [PR] [HUDI-6927] CDC file clean not work [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9841: URL: https://github.com/apache/hudi/pull/9841#issuecomment-1754300764 ## CI report: * 7cc75814a8d31fe885ce7538c963a40691d06a5c UNKNOWN * a2a9f0ee24ba017f4e0c6148d182df501cca3ad7 Azure:

Re: [PR] [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table [hudi]

2023-10-09 Thread via GitHub
bvaradar commented on PR #8143: URL: https://github.com/apache/hudi/pull/8143#issuecomment-1754300940 @xicm : Doing rebase of this PR. Once the test passes, will land this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1754300514 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * de2560dfe639845337ad6dd861e8cf980c2a3c3b Azure:

Re: [PR] [HUDI-6926] Disable DROP_PARTITION_COLUMNS when upsert MOR table [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9840: URL: https://github.com/apache/hudi/pull/9840#issuecomment-1754300698 ## CI report: * e8bb2bf52d648aa10ad684f6b5ee35ee75d28fcc Azure:

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9581: URL: https://github.com/apache/hudi/pull/9581#issuecomment-1754300024 ## CI report: * 391fc1099c8b51221a0448d8ac9d74e3013c1cd1 Azure:

Re: [PR] [HUDI-5911] SimpleTransactionDirectMarkerBasedDetectionStrategy can't work with none-partitioned table [hudi]

2023-10-09 Thread via GitHub
bvaradar commented on code in PR #8143: URL: https://github.com/apache/hudi/pull/8143#discussion_r1351340612 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/DirectMarkerTransactionManager.java: ## @@ -83,7 +83,7 @@ private static

[jira] [Created] (HUDI-6930) Eliminate the shouldLock flag for TimeGenerator

2023-10-09 Thread Danny Chen (Jira)
Danny Chen created HUDI-6930: Summary: Eliminate the shouldLock flag for TimeGenerator Key: HUDI-6930 URL: https://issues.apache.org/jira/browse/HUDI-6930 Project: Apache Hudi Issue Type:

Re: [PR] [HUDI-6927] CDC file clean not work [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9841: URL: https://github.com/apache/hudi/pull/9841#issuecomment-1754288912 ## CI report: * 7cc75814a8d31fe885ce7538c963a40691d06a5c UNKNOWN * a2a9f0ee24ba017f4e0c6148d182df501cca3ad7 Azure:

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9581: URL: https://github.com/apache/hudi/pull/9581#issuecomment-1754287454 ## CI report: * 391fc1099c8b51221a0448d8ac9d74e3013c1cd1 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [HUDI-5220] fix hive snapshot query add non hoodie paths file status [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #7206: URL: https://github.com/apache/hudi/pull/7206#issuecomment-1754281483 ## CI report: * 4abce05400396ee7019109306349e610a5e7ce71 Azure:

Re: [PR] [HUDI-6872] Test out of box schema evolution for deltastreamer [hudi]

2023-10-09 Thread via GitHub
the-other-tim-brown commented on code in PR #9743: URL: https://github.com/apache/hudi/pull/9743#discussion_r1351288017 ## hudi-common/src/main/java/org/apache/hudi/avro/AvroCastingGenericRecord.java: ## @@ -0,0 +1,259 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] [HUDI-6872] Test out of box schema evolution for deltastreamer [hudi]

2023-10-09 Thread via GitHub
the-other-tim-brown commented on code in PR #9743: URL: https://github.com/apache/hudi/pull/9743#discussion_r1351283672 ## hudi-common/src/main/java/org/apache/hudi/avro/AvroCastingGenericRecord.java: ## @@ -0,0 +1,259 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] [HUDI-6786] HoodieFileGroupReader integration [hudi]

2023-10-09 Thread via GitHub
codope commented on code in PR #9819: URL: https://github.com/apache/hudi/pull/9819#discussion_r1351254916 ## hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java: ## @@ -63,7 +64,7 @@ * @param The type of engine-specific record

Re: [I] [SUPPORT]does hudi[0.12.3] stream read support[-U]? I only got +I , +U , -D [hudi]

2023-10-09 Thread via GitHub
li-ang-666 commented on issue #9839: URL: https://github.com/apache/hudi/issues/9839#issuecomment-1754251032 and the read hudi is: create table enterprise( id DECIMAL(20, 0), op_ts TIMESTAMP(3), PRIMARY KEY (id) NOT ENFORCED )

Re: [I] [SUPPORT]does hudi[0.12.3] stream read support[-U]? I only got +I , +U , -D [hudi]

2023-10-09 Thread via GitHub
li-ang-666 commented on issue #9839: URL: https://github.com/apache/hudi/issues/9839#issuecomment-1754250121 here is write into hudi: CREATE TABLE source_table ( id DECIMAL(20, 0), op_ts TIMESTAMP(3) METADATA FROM 'value.ingestion-timestamp' VIRTUAL,

Re: [PR] [HUDI-6927] CDC file clean not work [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9841: URL: https://github.com/apache/hudi/pull/9841#issuecomment-1754233420 ## CI report: * 0c9df82ab63ae87f40d618ad5921449fab81aa00 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1754233082 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * 2b284abe79624c633d78e9f16e66dab74bc45329 Azure:

Re: [PR] [HUDI-6872] Test out of box schema evolution for deltastreamer [hudi]

2023-10-09 Thread via GitHub
the-other-tim-brown commented on code in PR #9743: URL: https://github.com/apache/hudi/pull/9743#discussion_r1351220251 ## hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java: ## @@ -661,6 +652,35 @@ private Pair>> fetchFromSourc return

Re: [PR] [HUDI-5220] fix hive snapshot query add non hoodie paths file status [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #7206: URL: https://github.com/apache/hudi/pull/7206#issuecomment-1754230235 ## CI report: * 4abce05400396ee7019109306349e610a5e7ce71 Azure:

Re: [PR] [HUDI-6927] CDC file clean not work [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9841: URL: https://github.com/apache/hudi/pull/9841#issuecomment-1754226907 ## CI report: * 0c9df82ab63ae87f40d618ad5921449fab81aa00 Azure:

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9776: URL: https://github.com/apache/hudi/pull/9776#issuecomment-1754226799 ## CI report: * 6b730068fa6ca60dfdd04f720334a49fa19a8b31 UNKNOWN * 2b284abe79624c633d78e9f16e66dab74bc45329 Azure:

Re: [PR] [HUDI-6927] CDC file clean not work [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9841: URL: https://github.com/apache/hudi/pull/9841#issuecomment-1754221180 ## CI report: * 0c9df82ab63ae87f40d618ad5921449fab81aa00 Azure:

Re: [I] [SUPPORT]does hudi[0.12.3] stream read support[-U]? I only got +I , +U , -D [hudi]

2023-10-09 Thread via GitHub
li-ang-666 commented on issue #9839: URL: https://github.com/apache/hudi/issues/9839#issuecomment-1754217048 > Did you enable the changelog mode? https://github.com/apache/hudi/assets/71430517/3585f89b-342d-412d-ad01-8f0c5c5fba49;> -- This is an automated message from the

Re: [I] [SUPPORT] Clarify the Multi Writer guarantee for duplicates [hudi]

2023-10-09 Thread via GitHub
xicm commented on issue #9831: URL: https://github.com/apache/hudi/issues/9831#issuecomment-1754203688 https://hudi.apache.org/docs/faq/#can-concurrent-inserts-cause-duplicates -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1351187636 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -216,6 +233,9 @@ protected List buildFileGroups(Stream baseFileS

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1351187636 ## hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java: ## @@ -216,6 +233,9 @@ protected List buildFileGroups(Stream baseFileS

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1351183761 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/rollback/TestMergeOnReadRollbackActionExecutor.java: ## @@ -126,21 +126,19 @@ public void

Re: [I] [SUPPORT]does hudi[0.12.3] stream read support[-U]? I only got +I , +U , -D [hudi]

2023-10-09 Thread via GitHub
li-ang-666 commented on issue #9839: URL: https://github.com/apache/hudi/issues/9839#issuecomment-1754193894 > Did you enable the changelog mode? yes,I set `changelog.enable=true` both in write and read -- This is an automated message from the Apache Git Service. To respond to the

[jira] [Created] (HUDI-6929) Make the default eager loading start time configurable for CompletionTimeQueryView

2023-10-09 Thread Danny Chen (Jira)
Danny Chen created HUDI-6929: Summary: Make the default eager loading start time configurable for CompletionTimeQueryView Key: HUDI-6929 URL: https://issues.apache.org/jira/browse/HUDI-6929 Project:

[jira] [Updated] (HUDI-6929) Make the default eager loading start time configurable for CompletionTimeQueryView

2023-10-09 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-6929: - Epic Link: HUDI-6640 > Make the default eager loading start time configurable for >

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1351135624 ## hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java: ## @@ -16,27 +16,33 @@ * limitations under the License. */

Re: [PR] [HUDI-6642] Use completion time based file slicing [hudi]

2023-10-09 Thread via GitHub
danny0405 commented on code in PR #9776: URL: https://github.com/apache/hudi/pull/9776#discussion_r1351133865 ## hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java: ## @@ -544,15 +543,40 @@ public static Option> getLatestLogVersion(FileSystem fs, P } /**

Re: [PR] [HUDI-6809] Optimizing the judgment of generating clustering plans [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9585: URL: https://github.com/apache/hudi/pull/9585#issuecomment-1754163412 ## CI report: * 1e4eb20f9bc3fad10ae7869c5132b671cbcbf344 Azure:

Re: [I] [SUPPORT]does hudi[0.12.3] stream read support[-U]? I only got +I , +U , -D [hudi]

2023-10-09 Thread via GitHub
danny0405 commented on issue #9839: URL: https://github.com/apache/hudi/issues/9839#issuecomment-1754159660 Did you enable the changelog mode? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [HUDI-6927] CDC file clean not work [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9841: URL: https://github.com/apache/hudi/pull/9841#issuecomment-1754064058 ## CI report: * 6c127303af21c9ff60ae4d210a8e648b9dd272ef Azure:

Re: [PR] [HUDI-6786] HoodieFileGroupReader integration [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9819: URL: https://github.com/apache/hudi/pull/9819#issuecomment-1754063999 ## CI report: * a4985db0ce22fb4b4f2518ed70bd96890024a08b UNKNOWN * 331e3ea538a42ac86c0208d348ec1432c3745373 Azure:

Re: [PR] [HUDI-6927] CDC file clean not work [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9841: URL: https://github.com/apache/hudi/pull/9841#issuecomment-1754058339 ## CI report: * 6c127303af21c9ff60ae4d210a8e648b9dd272ef Azure:

Re: [PR] [HUDI-6786] HoodieFileGroupReader integration [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9819: URL: https://github.com/apache/hudi/pull/9819#issuecomment-1754058285 ## CI report: * a4985db0ce22fb4b4f2518ed70bd96890024a08b UNKNOWN * 22fb8e8c4f28a18967d7fe924e145c759d39e2af Azure:

Re: [PR] [HUDI-6809] Optimizing the judgment of generating clustering plans [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9585: URL: https://github.com/apache/hudi/pull/9585#issuecomment-1754052496 ## CI report: * fd0791f4a32d638fde6c17583708607dc4017c65 Azure:

[jira] [Updated] (HUDI-6928) Support position based merging

2023-10-09 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Liu updated HUDI-6928: -- Epic Link: HUDI-6243 > Support position based merging > -- > > Key:

[jira] [Updated] (HUDI-6928) Support position based merging in HoodieFileGroupReader

2023-10-09 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Liu updated HUDI-6928: -- Summary: Support position based merging in HoodieFileGroupReader (was: Support position based merging) >

[jira] [Assigned] (HUDI-6786) Integrate FileGroupReader with NewHoodieParquetFileFormat for Spark MOR Snapshot Query

2023-10-09 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Liu reassigned HUDI-6786: - Assignee: Lin Liu (was: Jonathan Vexler) > Integrate FileGroupReader with NewHoodieParquetFileFormat

[jira] [Assigned] (HUDI-6928) Support position based merging

2023-10-09 Thread Lin Liu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Liu reassigned HUDI-6928: - Assignee: Lin Liu > Support position based merging > -- > >

[jira] [Created] (HUDI-6928) Support position based merging

2023-10-09 Thread Lin Liu (Jira)
Lin Liu created HUDI-6928: - Summary: Support position based merging Key: HUDI-6928 URL: https://issues.apache.org/jira/browse/HUDI-6928 Project: Apache Hudi Issue Type: New Feature

Re: [PR] [HUDI-6927] CDC file clean not work [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9841: URL: https://github.com/apache/hudi/pull/9841#issuecomment-1754002680 ## CI report: * 6c127303af21c9ff60ae4d210a8e648b9dd272ef Azure:

Re: [PR] [HUDI-6809] Optimizing the judgment of generating clustering plans [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9585: URL: https://github.com/apache/hudi/pull/9585#issuecomment-1754001876 ## CI report: * fd0791f4a32d638fde6c17583708607dc4017c65 Azure:

Re: [PR] [HUDI-6927] CDC file clean not work [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9841: URL: https://github.com/apache/hudi/pull/9841#issuecomment-1753984315 ## CI report: * 6c127303af21c9ff60ae4d210a8e648b9dd272ef UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[PR] [HUDI-6927] CDC file clean not work [hudi]

2023-10-09 Thread via GitHub
zhuanshenbsj1 opened a new pull request, #9841: URL: https://github.com/apache/hudi/pull/9841 ### Change Logs refer to https://github.com/apache/hudi/issues/9557 ### Impact _Describe any public API or user-facing feature change or any performance impact._ ###

[jira] [Created] (HUDI-6927) CDC file clean not work

2023-10-09 Thread zhuanshenbsj1 (Jira)
zhuanshenbsj1 created HUDI-6927: --- Summary: CDC file clean not work Key: HUDI-6927 URL: https://issues.apache.org/jira/browse/HUDI-6927 Project: Apache Hudi Issue Type: Improvement

Re: [PR] [HUDI-6923] Fixing bug with sanitization for rowSource [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9834: URL: https://github.com/apache/hudi/pull/9834#issuecomment-1753904489 ## CI report: * d28ebc812328746cb530a35db70df43e67c6ffc2 Azure:

Re: [PR] [HUDI-6926] Disable DROP_PARTITION_COLUMNS when upsert MOR table [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9840: URL: https://github.com/apache/hudi/pull/9840#issuecomment-1753462740 ## CI report: * e8bb2bf52d648aa10ad684f6b5ee35ee75d28fcc Azure:

Re: [PR] [HUDI-6926] Disable DROP_PARTITION_COLUMNS when upsert MOR table [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9840: URL: https://github.com/apache/hudi/pull/9840#issuecomment-1753411606 ## CI report: * e8bb2bf52d648aa10ad684f6b5ee35ee75d28fcc UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [HUDI-6923] Fixing bug with sanitization for rowSource [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9834: URL: https://github.com/apache/hudi/pull/9834#issuecomment-1753411489 ## CI report: * bd94b2cc68c5c339394bcaf9b077093eb6e9f18e Azure:

Re: [PR] [HUDI-6923] Fixing bug with sanitization for rowSource [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9834: URL: https://github.com/apache/hudi/pull/9834#issuecomment-1753400924 ## CI report: * bd94b2cc68c5c339394bcaf9b077093eb6e9f18e Azure:

[jira] [Updated] (HUDI-6926) Disable DROP_PARTITION_COLUMNS when upsert MOR table

2023-10-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6926: - Labels: pull-request-available (was: ) > Disable DROP_PARTITION_COLUMNS when upsert MOR table >

[PR] [HUDI-6926] Disable DROP_PARTITION_COLUMNS when upsert MOR table [hudi]

2023-10-09 Thread via GitHub
wecharyu opened a new pull request, #9840: URL: https://github.com/apache/hudi/pull/9840 ### Change Logs When merge the log files for MOR table, it can not generate partition values if we drop partition columns while upsert data. As a result, the updated rows in log files will get null

[jira] [Created] (HUDI-6926) Disable DROP_PARTITION_COLUMNS when upsert MOR table

2023-10-09 Thread Wechar (Jira)
Wechar created HUDI-6926: Summary: Disable DROP_PARTITION_COLUMNS when upsert MOR table Key: HUDI-6926 URL: https://issues.apache.org/jira/browse/HUDI-6926 Project: Apache Hudi Issue Type: Bug

Re: [PR] [HUDI-6923] Fixing bug with sanitization for rowSource [hudi]

2023-10-09 Thread via GitHub
harsh1231 commented on code in PR #9834: URL: https://github.com/apache/hudi/pull/9834#discussion_r1350564789 ## hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestSourceFormatAdapter.java: ## @@ -136,16 +146,23 @@ public void testJsonSanitization(String

Re: [PR] [HUDI-6923] Fixing bug with sanitization for rowSource [hudi]

2023-10-09 Thread via GitHub
the-other-tim-brown commented on code in PR #9834: URL: https://github.com/apache/hudi/pull/9834#discussion_r1350528910 ## hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestSourceFormatAdapter.java: ## @@ -136,16 +146,23 @@ public void

Re: [PR] [HUDI-6923] Fixing bug with sanitization for rowSource [hudi]

2023-10-09 Thread via GitHub
harsh1231 commented on code in PR #9834: URL: https://github.com/apache/hudi/pull/9834#discussion_r1350521264 ## hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestSourceFormatAdapter.java: ## @@ -136,16 +146,23 @@ public void testJsonSanitization(String

Re: [PR] [HUDI-6923] Fixing bug with sanitization for rowSource [hudi]

2023-10-09 Thread via GitHub
the-other-tim-brown commented on code in PR #9834: URL: https://github.com/apache/hudi/pull/9834#discussion_r1350512605 ## hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestSourceFormatAdapter.java: ## @@ -136,16 +146,23 @@ public void

Re: [PR] [HUDI-6923] Fixing bug with sanitization for rowSource [hudi]

2023-10-09 Thread via GitHub
the-other-tim-brown commented on code in PR #9834: URL: https://github.com/apache/hudi/pull/9834#discussion_r1350509191 ## hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestSourceFormatAdapter.java: ## @@ -136,16 +146,23 @@ public void

Re: [I] [SUPPORT]- Delete support in Hudi Flink [hudi]

2023-10-09 Thread via GitHub
shenbachand commented on issue #9787: URL: https://github.com/apache/hudi/issues/9787#issuecomment-1753224251 @danny0405 - Sorry for late reply. Yes my job is streaming to write the data from kafka to S3 -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9581: URL: https://github.com/apache/hudi/pull/9581#issuecomment-1753211609 ## CI report: * 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN * 391fc1099c8b51221a0448d8ac9d74e3013c1cd1 Azure:

[I] [SUPPORT] Hudi MERGE INTO on Glue fails when using functions such as (filter, zip_with) on array of structs [hudi]

2023-10-09 Thread via GitHub
rita-ihnatsyeva opened a new issue, #9838: URL: https://github.com/apache/hudi/issues/9838 - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? yes **Describe the problem you faced** Prerequisites: 1. I have a permanent hudi table in AWS Glue 2.

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9581: URL: https://github.com/apache/hudi/pull/9581#issuecomment-1753135409 ## CI report: * 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN * bda7b26fbc477b408e8658eb6e52badeb5c653e0 Azure:

Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9581: URL: https://github.com/apache/hudi/pull/9581#issuecomment-1753117528 ## CI report: * 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN * bda7b26fbc477b408e8658eb6e52badeb5c653e0 Azure:

Re: [PR] [HUDI-6925] Do not list all partitions for 'alter table drop partition' [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9837: URL: https://github.com/apache/hudi/pull/9837#issuecomment-1753009296 ## CI report: * 8128d5cbc6075f2b2fd7a8b7bf5e940d6493a66f Azure:

Re: [PR] [MINOR] scanner should closed properly in HoodieComactor [hudi]

2023-10-09 Thread via GitHub
xuzifu666 commented on PR #9835: URL: https://github.com/apache/hudi/pull/9835#issuecomment-1752978510 cc @danny0405 @yihua -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] [MINOR] scanner should closed properly in HoodieComactor [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9835: URL: https://github.com/apache/hudi/pull/9835#issuecomment-1752923408 ## CI report: * 0e564fbc0b7e5e10e6a61bff77014de0f33b82bb Azure:

Re: [I] [BUG] S3 Deltastreamer: Block has already been inflated [hudi]

2023-10-09 Thread via GitHub
envomp commented on issue #6428: URL: https://github.com/apache/hudi/issues/6428#issuecomment-1752836684 Hey @ad1happy2go We have the following s3a configurations: ``` spark.hadoop.fs.s3a.path.style.access: true spark.hadoop.fs.s3a.threads.max: 64

Re: [I] [SUPPORT] Clarify the Multi Writer guarantee for duplicates [hudi]

2023-10-09 Thread via GitHub
istreeter commented on issue #9831: URL: https://github.com/apache/hudi/issues/9831#issuecomment-1752826301 Thank you @xicm for sharing the link to RFC-66. This was enlightening for how the problem will be addressed in future. But for now keeping the conversation about the current

Re: [PR] [HUDI-6925] Do not list all partitions for 'alter table drop partition' [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9837: URL: https://github.com/apache/hudi/pull/9837#issuecomment-1752817160 ## CI report: * 8128d5cbc6075f2b2fd7a8b7bf5e940d6493a66f Azure:

Re: [PR] [HUDI-6924] Fix hoodie table config not wok in table properties [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9836: URL: https://github.com/apache/hudi/pull/9836#issuecomment-1752817066 ## CI report: * ec64b99d8525d88a1844e434000d83d06e74fe75 Azure:

Re: [PR] [HUDI-6832] Ensure other table services with correct path are not affect… [hudi]

2023-10-09 Thread via GitHub
leesf merged PR #9652: URL: https://github.com/apache/hudi/pull/9652 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[hudi] branch master updated: [HUDI-6832] Ensure other table services with correct path are not affected by table with wrong path. (#9652)

2023-10-09 Thread leesf
This is an automated email from the ASF dual-hosted git repository. leesf pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 8e434b816dc [HUDI-6832] Ensure other table

Re: [I] [SUPPORT] Parquet files got cleaned up even when cleaning operation failed hence leading to subsequent failed clustering and cleaning [hudi]

2023-10-09 Thread via GitHub
adityaverma1997 commented on issue #9257: URL: https://github.com/apache/hudi/issues/9257#issuecomment-1752778848 @ad1happy2go Any update on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [HUDI-6925] Do not list all partitions for 'alter table drop partition' [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9837: URL: https://github.com/apache/hudi/pull/9837#issuecomment-1752757006 ## CI report: * 8128d5cbc6075f2b2fd7a8b7bf5e940d6493a66f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

Re: [PR] [HUDI-6924] Fix hoodie table config not wok in table properties [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9836: URL: https://github.com/apache/hudi/pull/9836#issuecomment-1752744004 ## CI report: * ec64b99d8525d88a1844e434000d83d06e74fe75 Azure:

Re: [PR] [HUDI-6923] Fixing bug with sanitization for rowSource [hudi]

2023-10-09 Thread via GitHub
hudi-bot commented on PR #9834: URL: https://github.com/apache/hudi/pull/9834#issuecomment-1752743925 ## CI report: * bd94b2cc68c5c339394bcaf9b077093eb6e9f18e Azure:

Re: [I] [SUPPORT] Trino can't read tables created by Flink Hudi conector [hudi]

2023-10-09 Thread via GitHub
galadrielwithlaptop commented on issue #9435: URL: https://github.com/apache/hudi/issues/9435#issuecomment-1752691995 Steps to repro: 1. We need to add calcite-core dependency in the server classpath. 2. We need to follow this doc:

Re: [PR] [HUDI-6925] Do not list all partitions for 'alter table drop partition' [hudi]

2023-10-09 Thread via GitHub
stream2000 commented on PR #9837: URL: https://github.com/apache/hudi/pull/9837#issuecomment-1752676594 @leesf @boneanxs Hi, could you help review this PR? Thanks~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[jira] [Updated] (HUDI-6925) Do not list all partitions when deleting partition

2023-10-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6925: - Labels: pull-request-available (was: ) > Do not list all partitions when deleting partition >

[PR] [HUDI-6925] Do not list all partitions for 'alter table drop partition' [hudi]

2023-10-09 Thread via GitHub
stream2000 opened a new pull request, #9837: URL: https://github.com/apache/hudi/pull/9837 ### Change Logs Currently we will list all partitions paths when calling 'alter table drop partition' to check if the table is hive style partitions and URL encoded. Listing all partitions

  1   2   >