[GitHub] [hudi] michael1991 opened a new issue, #8092: [SUPPORT] Spell Mistake on Hudi Configurations Doc

2023-03-02 Thread via GitHub
michael1991 opened a new issue, #8092: URL: https://github.com/apache/hudi/issues/8092 > hoodie.datasource.write.partitionpath.field > Partition path field. Value to be used at the partitionPath component of HoodieKey. Actual value ontained by invoking .toString() > Default Value: N/A

[GitHub] [hudi] hudi-bot commented on pull request #8070: [HUDI-4372] Enable metadata table by default for flink

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8070: URL: https://github.com/apache/hudi/pull/8070#issuecomment-1453106546 ## CI report: * 39fcab73829e0dfc830c11c80bb852b5f95deaa9 Azure:

[GitHub] [hudi] michael1991 commented on issue #8075: [SUPPORT] Issues on Writing data to GCS

2023-03-02 Thread via GitHub
michael1991 commented on issue #8075: URL: https://github.com/apache/hudi/issues/8075#issuecomment-1453102780 > can you try setting up this config > > ``` > --conf 'spark.hadoop.fs.gs.outputstream.pipe. > type=NIO_CHANNEL_PIPE' > ``` @nsivabalan Hi, thanks for

[GitHub] [hudi] vinaykv1991 commented on pull request #8091: [DOCS] Update gcp_bigquery.md

2023-03-02 Thread via GitHub
vinaykv1991 commented on PR #8091: URL: https://github.com/apache/hudi/pull/8091#issuecomment-1453091543 @bhasudha @nfarah86 @codope please review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [hudi] vinaykv1991 opened a new pull request, #8091: Update gcp_bigquery.md

2023-03-02 Thread via GitHub
vinaykv1991 opened a new pull request, #8091: URL: https://github.com/apache/hudi/pull/8091 Why changes - In what cases the integration works is clear, finding elaboration of that cases can be made easier with links. What are the Changes - Added links to understand in what cases HUDI Big

[GitHub] [hudi] hudi-bot commented on pull request #8070: [HUDI-4372] Enable metadata table by default for flink

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8070: URL: https://github.com/apache/hudi/pull/8070#issuecomment-1453071140 ## CI report: * 39fcab73829e0dfc830c11c80bb852b5f95deaa9 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8041: [HUDI-5847] Add support for multiple metric reporters and metric labels

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8041: URL: https://github.com/apache/hudi/pull/8041#issuecomment-1453064225 ## CI report: * 6995a948f49fadbec59748c4728a2beef6072b36 UNKNOWN * d04edd5df0b50036aa9ef5175fa3f46f0b0f4c6f Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8041: [HUDI-5847] Add support for multiple metric reporters and metric labels

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8041: URL: https://github.com/apache/hudi/pull/8041#issuecomment-1453058343 ## CI report: * 6995a948f49fadbec59748c4728a2beef6072b36 UNKNOWN * d04edd5df0b50036aa9ef5175fa3f46f0b0f4c6f Azure:

[GitHub] [hudi] BalaMahesh commented on pull request #7687: [HUDI-5606] Update to handle deletes in postgres debezium

2023-03-02 Thread via GitHub
BalaMahesh commented on PR #7687: URL: https://github.com/apache/hudi/pull/7687#issuecomment-1453041242 @nsivabalan / @rmahindra123 / @the-other-tim-brown can you please review this pr, we are running this patch in our prod pipelines, without merging this it would be difficult for us to

[jira] [Updated] (HUDI-5606) Update to handle deletes in postgres debezium

2023-03-02 Thread Bala Mahesh Jampani (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bala Mahesh Jampani updated HUDI-5606: -- Description: We have onboarded our postgres tables to hudi via debezium and Kafka. It

[jira] [Updated] (HUDI-5606) Update to handle deletes in postgres debezium

2023-03-02 Thread Bala Mahesh Jampani (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bala Mahesh Jampani updated HUDI-5606: -- Description: We have onboarded our postgres tables to hudi via debezium and Kafka. It

[jira] [Assigned] (HUDI-5606) Update to handle deletes in postgres debezium

2023-03-02 Thread Bala Mahesh Jampani (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bala Mahesh Jampani reassigned HUDI-5606: - Assignee: Bala Mahesh Jampani > Update to handle deletes in postgres debezium >

[GitHub] [hudi] hudi-bot commented on pull request #8088: [HUDI-5873] The pending compactions of dataset table should not block…

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8088: URL: https://github.com/apache/hudi/pull/8088#issuecomment-1453004682 ## CI report: * c65842899078697c5c5ff647e89f7cf918531f8d Azure:

[hudi] branch master updated (6b178ce978c -> 8e7524574ce)

2023-03-02 Thread vbalaji
This is an automated email from the ASF dual-hosted git repository. vbalaji pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 6b178ce978c [HUDI-4442] [HUDI-5001] Sanitize JsonConversion and RowSource (#8010) add 8e7524574ce [HUDI-5728]

[GitHub] [hudi] bvaradar merged pull request #7891: [HUDI-5728] HoodieTimelineArchiver archives the latest instant before inflight replacecommit

2023-03-02 Thread via GitHub
bvaradar merged PR #7891: URL: https://github.com/apache/hudi/pull/7891 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] bvaradar commented on pull request #7891: [HUDI-5728] HoodieTimelineArchiver archives the latest instant before inflight replacecommit

2023-03-02 Thread via GitHub
bvaradar commented on PR #7891: URL: https://github.com/apache/hudi/pull/7891#issuecomment-1452976927 @SteNicholas Makes sense. Thanks for the patience. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hudi] hudi-bot commented on pull request #8086: [HUDI-5872] Abstraction for DeltaSyncService and DeltaStreamerMetrics

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8086: URL: https://github.com/apache/hudi/pull/8086#issuecomment-1452962332 ## CI report: * 9f72cf006102bcb34a24935f5da9263d4d98f537 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8070: [HUDI-4372] Enable metadata table by default for flink

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8070: URL: https://github.com/apache/hudi/pull/8070#issuecomment-1452956970 ## CI report: * 39fcab73829e0dfc830c11c80bb852b5f95deaa9 Azure:

[GitHub] [hudi] waitingF commented on pull request #7811: [HUDI-5518] Support canal-json for HoodieDeltaStreamer

2023-03-02 Thread via GitHub
waitingF commented on PR #7811: URL: https://github.com/apache/hudi/pull/7811#issuecomment-1452929344 hi team, any progress on this? may I continue this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs

2023-03-02 Thread via GitHub
hudi-bot commented on PR #7901: URL: https://github.com/apache/hudi/pull/7901#issuecomment-1452915990 ## CI report: * 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN * d8046498c1811e9e0a446cef557514007f47 Azure:

[GitHub] [hudi] nsivabalan commented on issue #8065: [SUPPORT] Deltastreamer AvroKafka Schema Evolution transiently failing in --continuous mode

2023-03-02 Thread via GitHub
nsivabalan commented on issue #8065: URL: https://github.com/apache/hudi/issues/8065#issuecomment-1452897668 hey @danielfordfc I guess we have some hunch on whats going on. if you have some time and willing to contribute, let me know. The issue is. getSourceScheme in case

[jira] [Closed] (HUDI-5817) Fix async indexer metadata writer to avoid eager rollback / cleaning

2023-03-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-5817. --- Resolution: Fixed > Fix async indexer metadata writer to avoid eager rollback / cleaning >

[jira] [Updated] (HUDI-5666) Support custom compaction strategy to compact files partition in MDT aggressively

2023-03-02 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-5666: -- Epic Link: HUDI-1292 > Support custom compaction strategy to compact files partition in

[jira] [Updated] (HUDI-5694) Avoid unnecessary file system parsing to initialize a metadata for a new data table

2023-03-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5694: Epic Link: HUDI-1292 > Avoid unnecessary file system parsing to initialize a metadata for a new data >

[GitHub] [hudi] nsivabalan commented on issue #8075: [SUPPORT] Issues on Writing data to GCS

2023-03-02 Thread via GitHub
nsivabalan commented on issue #8075: URL: https://github.com/apache/hudi/issues/8075#issuecomment-1452891504 Added a faq on this https://github.com/apache/hudi/pull/8090 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [hudi] nsivabalan closed issue #8075: [SUPPORT] Issues on Writing data to GCS

2023-03-02 Thread via GitHub
nsivabalan closed issue #8075: [SUPPORT] Issues on Writing data to GCS URL: https://github.com/apache/hudi/issues/8075 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[jira] [Updated] (HUDI-5768) Fail to read metadata table in Spark Datasource

2023-03-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5768: Epic Link: HUDI-1292 > Fail to read metadata table in Spark Datasource >

[GitHub] [hudi] nsivabalan opened a new pull request, #8090: [DOCS] Adding faq on GCS issue w/ writes

2023-03-02 Thread via GitHub
nsivabalan opened a new pull request, #8090: URL: https://github.com/apache/hudi/pull/8090 ### Change Logs Adding faq on GCS issue w/ writes ### Impact _Describe any public API or user-facing feature change or any performance impact._ ### Risk level (write none,

[jira] [Updated] (HUDI-5817) Fix async indexer metadata writer to avoid eager rollback / cleaning

2023-03-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5817: Epic Link: HUDI-1292 > Fix async indexer metadata writer to avoid eager rollback / cleaning >

[jira] [Updated] (HUDI-5863) Fix HoodieMetadataFileSystemView serving stale view at the timeline server

2023-03-02 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-5863: Epic Link: HUDI-1292 > Fix HoodieMetadataFileSystemView serving stale view at the timeline server >

[GitHub] [hudi] nsivabalan closed pull request #8089: [DOCS] Adding faq on GCS write failure

2023-03-02 Thread via GitHub
nsivabalan closed pull request #8089: [DOCS] Adding faq on GCS write failure URL: https://github.com/apache/hudi/pull/8089 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] nsivabalan opened a new pull request, #8089: [DOCS] Adding faq on GCS write failure

2023-03-02 Thread via GitHub
nsivabalan opened a new pull request, #8089: URL: https://github.com/apache/hudi/pull/8089 ### Change Logs Adding a faq on an issue w/ GCS. ### Impact _Describe any public API or user-facing feature change or any performance impact._ ### Risk level (write none,

[GitHub] [hudi] michael1991 commented on issue #7595: [SUPPORT] Hudi Clean and Delta commits taking ~50 mins to finish frequently

2023-03-02 Thread via GitHub
michael1991 commented on issue #7595: URL: https://github.com/apache/hudi/issues/7595#issuecomment-1452883556 Same issue faced here, but we use Spark to append MOR tables with async cleaning. A lot of warning messages came into log file, any good ideas more ? -- This is an automated

[GitHub] [hudi] nsivabalan commented on issue #8075: [SUPPORT] Issues on Writing data to GCS

2023-03-02 Thread via GitHub
nsivabalan commented on issue #8075: URL: https://github.com/apache/hudi/issues/8075#issuecomment-1452881688 can you try setting up this config ``` --conf 'spark.hadoop.fs.gs.outputstream.pipe. type=NIO_CHANNEL_PIPE' ``` -- This is an automated message from the Apache Git

[GitHub] [hudi] hudi-bot commented on pull request #8088: [HUDI-5873] The pending compactions of dataset table should not block…

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8088: URL: https://github.com/apache/hudi/pull/8088#issuecomment-1452880303 ## CI report: * c65842899078697c5c5ff647e89f7cf918531f8d Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8088: [HUDI-5873] The pending compactions of dataset table should not block…

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8088: URL: https://github.com/apache/hudi/pull/8088#issuecomment-1452876024 ## CI report: * c65842899078697c5c5ff647e89f7cf918531f8d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-5873) The pending compactions of dataset table should not block MDT compaction

2023-03-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5873: - Labels: pull-request-available (was: ) > The pending compactions of dataset table should not

[GitHub] [hudi] danny0405 opened a new pull request, #8088: [HUDI-5873] The pending compactions of dataset table should not block…

2023-03-02 Thread via GitHub
danny0405 opened a new pull request, #8088: URL: https://github.com/apache/hudi/pull/8088 … MDT compaction ### Change Logs Adjust the MDT compaction strategy to not blocked by DT pending compactions. ### Impact Could reduce the metadata small files significantly

[GitHub] [hudi] hudi-bot commented on pull request #8086: [HUDI-5872] Abstraction for DeltaSyncService and DeltaStreamerMetrics

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8086: URL: https://github.com/apache/hudi/pull/8086#issuecomment-1452822280 ## CI report: * 006ee4a7eee6e217f060b210de92838312384f98 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8086: [HUDI-5872] Abstraction for DeltaSyncService and DeltaStreamerMetrics

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8086: URL: https://github.com/apache/hudi/pull/8086#issuecomment-1452817203 ## CI report: * 006ee4a7eee6e217f060b210de92838312384f98 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8070: [HUDI-4372] Enable metadata table by default for flink

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8070: URL: https://github.com/apache/hudi/pull/8070#issuecomment-1452817117 ## CI report: * Unknown: [CANCELED](TBD) * 39fcab73829e0dfc830c11c80bb852b5f95deaa9 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8082: [HUDI-5868] Upgrade Spark to 3.3.2

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8082: URL: https://github.com/apache/hudi/pull/8082#issuecomment-1452811793 ## CI report: * 61dda6da1e111009d968f3af1735f56b43181be7 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8070: [HUDI-4372] Enable metadata table by default for flink

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8070: URL: https://github.com/apache/hudi/pull/8070#issuecomment-1452811718 ## CI report: * Unknown: [CANCELED](TBD) * 39fcab73829e0dfc830c11c80bb852b5f95deaa9 UNKNOWN Bot commands @hudi-bot supports the following commands: -

[GitHub] [hudi] xiarixiaoyao commented on pull request #8084: [HUDI-5870] Fix some comments about column type change rules

2023-03-02 Thread via GitHub
xiarixiaoyao commented on PR #8084: URL: https://github.com/apache/hudi/pull/8084#issuecomment-1452807905 @mapleFU Thank you for your contribution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [hudi] 1032851561 opened a new issue, #8087: [SUPPORT] split_reader don't checkpoint before consuming all splits

2023-03-02 Thread via GitHub
1032851561 opened a new issue, #8087: URL: https://github.com/apache/hudi/issues/8087 **split_reader don't checkpoint before consuming all splits** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get

[jira] [Created] (HUDI-5874) Refactor DeltaStreamer config to use HoodieIngestionConfig

2023-03-02 Thread Raymond Xu (Jira)
Raymond Xu created HUDI-5874: Summary: Refactor DeltaStreamer config to use HoodieIngestionConfig Key: HUDI-5874 URL: https://issues.apache.org/jira/browse/HUDI-5874 Project: Apache Hudi Issue

[jira] [Created] (HUDI-5873) The pending compactions of dataset table should not block MDT compaction

2023-03-02 Thread Danny Chen (Jira)
Danny Chen created HUDI-5873: Summary: The pending compactions of dataset table should not block MDT compaction Key: HUDI-5873 URL: https://issues.apache.org/jira/browse/HUDI-5873 Project: Apache Hudi

[GitHub] [hudi] danny0405 commented on a diff in pull request #8070: [HUDI-4372] Enable metadata table by default for flink

2023-03-02 Thread via GitHub
danny0405 commented on code in PR #8070: URL: https://github.com/apache/hudi/pull/8070#discussion_r1122901308 ## packaging/bundle-validation/validate.sh: ## @@ -148,7 +148,7 @@ test_flink_bundle() { export HADOOP_CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath)

[GitHub] [hudi] danny0405 commented on pull request #8070: [HUDI-4372] Enable metadata table by default for flink

2023-03-02 Thread via GitHub
danny0405 commented on PR #8070: URL: https://github.com/apache/hudi/pull/8070#issuecomment-1452786258 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] hudi-bot commented on pull request #8082: [HUDI-5868] Upgrade Spark to 3.3.2

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8082: URL: https://github.com/apache/hudi/pull/8082#issuecomment-1452779602 ## CI report: * fb4c221c771d36a04ebf6bffbb9e993e4b32f4bf Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8082: [HUDI-5868] Upgrade Spark to 3.3.2

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8082: URL: https://github.com/apache/hudi/pull/8082#issuecomment-1452775663 ## CI report: * fb4c221c771d36a04ebf6bffbb9e993e4b32f4bf Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8086: [HUDI-5872] Abstraction for DeltaSyncService and DeltaStreamerMetrics

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8086: URL: https://github.com/apache/hudi/pull/8086#issuecomment-1452769837 ## CI report: * 006ee4a7eee6e217f060b210de92838312384f98 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8082: [HUDI-5868] Upgrade Spark to 3.3.2

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8082: URL: https://github.com/apache/hudi/pull/8082#issuecomment-1452769806 ## CI report: * fb4c221c771d36a04ebf6bffbb9e993e4b32f4bf Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs

2023-03-02 Thread via GitHub
hudi-bot commented on PR #7901: URL: https://github.com/apache/hudi/pull/7901#issuecomment-1452769509 ## CI report: * 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN * b0ddb5d1365ec41ce3890fe33a1aecba4a506b40 Azure:

[GitHub] [hudi] rahil-c commented on pull request #8082: [HUDI-5868] Upgrade Spark to 3.3.2

2023-03-02 Thread via GitHub
rahil-c commented on PR #8082: URL: https://github.com/apache/hudi/pull/8082#issuecomment-1452764196 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs

2023-03-02 Thread via GitHub
hudi-bot commented on PR #7901: URL: https://github.com/apache/hudi/pull/7901#issuecomment-1452733368 ## CI report: * 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN * b0ddb5d1365ec41ce3890fe33a1aecba4a506b40 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs

2023-03-02 Thread via GitHub
hudi-bot commented on PR #7901: URL: https://github.com/apache/hudi/pull/7901#issuecomment-1452686051 ## CI report: * 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN * b0ddb5d1365ec41ce3890fe33a1aecba4a506b40 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8086: [HUDI-5872] Abstraction for DeltaSyncService and DeltaStreamerMetrics

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8086: URL: https://github.com/apache/hudi/pull/8086#issuecomment-1452600034 ## CI report: * 006ee4a7eee6e217f060b210de92838312384f98 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8086: [HUDI-5872] Abstraction for DeltaSyncService and DeltaStreamerMetrics

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8086: URL: https://github.com/apache/hudi/pull/8086#issuecomment-1452592964 ## CI report: * 006ee4a7eee6e217f060b210de92838312384f98 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-5872) Abstraction for DeltaSyncService and DeltaStreamerMetrics

2023-03-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5872: - Labels: pull-request-available (was: ) > Abstraction for DeltaSyncService and

[GitHub] [hudi] xushiyan opened a new pull request, #8086: [HUDI-5872] Abstraction for DeltaSyncService and DeltaStreamerMetrics

2023-03-02 Thread via GitHub
xushiyan opened a new pull request, #8086: URL: https://github.com/apache/hudi/pull/8086 ### Change Logs Add abstraction classes for DeltaSyncService and DeltaStreamerMetrics - HoodieIngestionService - HoodieIngestionMetrics ### Impact NA ### Risk

[jira] [Updated] (HUDI-5872) Abstraction for DeltaSyncService and DeltaStreamerMetrics

2023-03-02 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-5872: - Sprint: Sprint 2023-02-14 > Abstraction for DeltaSyncService and DeltaStreamerMetrics >

[jira] [Created] (HUDI-5872) Abstraction for DeltaSyncService and DeltaStreamerMetrics

2023-03-02 Thread Raymond Xu (Jira)
Raymond Xu created HUDI-5872: Summary: Abstraction for DeltaSyncService and DeltaStreamerMetrics Key: HUDI-5872 URL: https://issues.apache.org/jira/browse/HUDI-5872 Project: Apache Hudi Issue

[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs

2023-03-02 Thread via GitHub
hudi-bot commented on PR #7901: URL: https://github.com/apache/hudi/pull/7901#issuecomment-1452525298 ## CI report: * 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN * 3d36b5c0206e3cab5d1551f22ae41082f3cc7354 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7901: [HUDI-5665] Adding support to re-use table configs

2023-03-02 Thread via GitHub
hudi-bot commented on PR #7901: URL: https://github.com/apache/hudi/pull/7901#issuecomment-1452517030 ## CI report: * 1318abbb3f3e93f9e2f2ed20d3fc543de0ef278c UNKNOWN * 3d36b5c0206e3cab5d1551f22ae41082f3cc7354 Azure:

[jira] [Created] (HUDI-5871) Bootstrap does not work with partitions with /

2023-03-02 Thread Jonathan Vexler (Jira)
Jonathan Vexler created HUDI-5871: - Summary: Bootstrap does not work with partitions with / Key: HUDI-5871 URL: https://issues.apache.org/jira/browse/HUDI-5871 Project: Apache Hudi Issue

[GitHub] [hudi] tatiana-rackspace opened a new issue, #8085: [SUPPORT] deltacommit triggering criteria

2023-03-02 Thread via GitHub
tatiana-rackspace opened a new issue, #8085: URL: https://github.com/apache/hudi/issues/8085 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at

[hudi] branch master updated: [HUDI-4442] [HUDI-5001] Sanitize JsonConversion and RowSource (#8010)

2023-03-02 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 6b178ce978c [HUDI-4442] [HUDI-5001] Sanitize

[GitHub] [hudi] nsivabalan merged pull request #8010: [HUDI-4442] [HUDI-5001] Sanitize JsonConversion and RowSource

2023-03-02 Thread via GitHub
nsivabalan merged PR #8010: URL: https://github.com/apache/hudi/pull/8010 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] jonvex commented on pull request #8010: [HUDI-4442] [HUDI-5001] Sanitize JsonConversion and RowSource

2023-03-02 Thread via GitHub
jonvex commented on PR #8010: URL: https://github.com/apache/hudi/pull/8010#issuecomment-1452006855 OK CI has passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] hudi-bot commented on pull request #8084: [HUDI-5870] Fix some comments about column type change rules

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8084: URL: https://github.com/apache/hudi/pull/8084#issuecomment-1451986035 ## CI report: * 1ddf2e9d22907e27807e0a0487032b2126a0 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8070: [HUDI-4372] Enable metadata table by default for flink

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8070: URL: https://github.com/apache/hudi/pull/8070#issuecomment-1451879131 ## CI report: * 73d21c4a8215d417f8a5231f76f24eacf34cc6ef Azure:

[GitHub] [hudi] voonhous commented on issue #8071: [SUPPORT]How to improve the speed of Flink writing to hudi ?

2023-03-02 Thread via GitHub
voonhous commented on issue #8071: URL: https://github.com/apache/hudi/issues/8071#issuecomment-1451874410 You are right that there are no Flink configs for it. What i did in the past was configure it using: `hoodie.parquet.compression.codec`. You can verify if whether the configuration

[GitHub] [hudi] soumilshah1995 commented on issue #8030: [SUPPORT] Stored procedure for converting smaller files into larger files for COW table type

2023-03-02 Thread via GitHub
soumilshah1995 commented on issue #8030: URL: https://github.com/apache/hudi/issues/8030#issuecomment-1451834511 Good Morning Team Could we have some follow up on this ticket -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] soumilshah1995 commented on issue #8040: [SUPPORT] Getting error when writing into MOR HUDI table if schema changed (datatype changed / column dropped)

2023-03-02 Thread via GitHub
soumilshah1995 commented on issue #8040: URL: https://github.com/apache/hudi/issues/8040#issuecomment-1451832086 Good Morning Can some one Please provide some updates regarding this tickets ? -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] hudi-bot commented on pull request #7895: [HUDI-5736] Common de-coupling column drop flag and schema validation flag

2023-03-02 Thread via GitHub
hudi-bot commented on PR #7895: URL: https://github.com/apache/hudi/pull/7895#issuecomment-1451827599 ## CI report: * 3b6dbf0bcc7059a7dc4a5132bf45d0d1451d7fb0 UNKNOWN * af0c75f62601109b018a20520b652affbbd19dcd Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8084: [HUDI-5870] Fix some comments about column type change rules

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8084: URL: https://github.com/apache/hudi/pull/8084#issuecomment-1451797979 ## CI report: * 1ddf2e9d22907e27807e0a0487032b2126a0 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8041: [HUDI-5847] Add support for multiple metric reporters and metric labels

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8041: URL: https://github.com/apache/hudi/pull/8041#issuecomment-1451797447 ## CI report: * 6995a948f49fadbec59748c4728a2beef6072b36 UNKNOWN * d04edd5df0b50036aa9ef5175fa3f46f0b0f4c6f Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8084: [HUDI-5870] Fix some comments about column type change rules

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8084: URL: https://github.com/apache/hudi/pull/8084#issuecomment-1451746802 ## CI report: * 1ddf2e9d22907e27807e0a0487032b2126a0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[jira] [Updated] (HUDI-5870) Fix some comments about column type change rules

2023-03-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-5870: - Labels: pull-request-available (was: ) > Fix some comments about column type change rules >

[GitHub] [hudi] mapleFU opened a new pull request, #8084: [HUDI-5870] Fix some comments about column type change rules

2023-03-02 Thread via GitHub
mapleFU opened a new pull request, #8084: URL: https://github.com/apache/hudi/pull/8084 ### Change Logs This patch didn't change any logic. It just update the comment for schema evolution type change. ### Impact No ### Risk level (write none, low medium or high

[jira] [Created] (HUDI-5870) Fix some comments about column type change rules

2023-03-02 Thread Xuwei Fu (Jira)
Xuwei Fu created HUDI-5870: -- Summary: Fix some comments about column type change rules Key: HUDI-5870 URL: https://issues.apache.org/jira/browse/HUDI-5870 Project: Apache Hudi Issue Type:

[GitHub] [hudi] hudi-bot commented on pull request #8070: [HUDI-4372] Enable metadata table by default for flink

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8070: URL: https://github.com/apache/hudi/pull/8070#issuecomment-1451680496 ## CI report: * d1c82b8e47d655ce9af03529c70263ba2ef6659a Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7978: [HUDI-5812] Optimize the data size check in HoodieBaseParquetWriter

2023-03-02 Thread via GitHub
hudi-bot commented on PR #7978: URL: https://github.com/apache/hudi/pull/7978#issuecomment-1451680016 ## CI report: * daa56e1dc5863d8b759b0304534badc0d3594e43 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8070: [HUDI-4372] Enable metadata table by default for flink

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8070: URL: https://github.com/apache/hudi/pull/8070#issuecomment-1451670561 ## CI report: * d1c82b8e47d655ce9af03529c70263ba2ef6659a Azure:

[GitHub] [hudi] danny0405 commented on a diff in pull request #8070: [HUDI-4372] Enable metadata table by default for flink

2023-03-02 Thread via GitHub
danny0405 commented on code in PR #8070: URL: https://github.com/apache/hudi/pull/8070#discussion_r1122901308 ## packaging/bundle-validation/validate.sh: ## @@ -148,7 +148,7 @@ test_flink_bundle() { export HADOOP_CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath)

[GitHub] [hudi] hudi-bot commented on pull request #8041: [HUDI-5847] Add support for multiple metric reporters and metric labels

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8041: URL: https://github.com/apache/hudi/pull/8041#issuecomment-1451658797 ## CI report: * e453f39e2e3f48eabcb1470922573b07d4ed486d Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7895: [HUDI-5736] Common de-coupling column drop flag and schema validation flag

2023-03-02 Thread via GitHub
hudi-bot commented on PR #7895: URL: https://github.com/apache/hudi/pull/7895#issuecomment-1451658240 ## CI report: * 3b6dbf0bcc7059a7dc4a5132bf45d0d1451d7fb0 UNKNOWN * c9d463794567d596503a7f9519325b88aa768f26 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8041: [HUDI-5847] Add support for multiple metric reporters and metric labels

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8041: URL: https://github.com/apache/hudi/pull/8041#issuecomment-1451648879 ## CI report: * e453f39e2e3f48eabcb1470922573b07d4ed486d Azure:

[GitHub] [hudi] hudi-bot commented on pull request #8041: [HUDI-5847] Add support for multiple metric reporters and metric labels

2023-03-02 Thread via GitHub
hudi-bot commented on PR #8041: URL: https://github.com/apache/hudi/pull/8041#issuecomment-1451591128 ## CI report: * e453f39e2e3f48eabcb1470922573b07d4ed486d Azure:

[GitHub] [hudi] SteNicholas commented on pull request #7891: [HUDI-5728] HoodieTimelineArchiver archives the latest instant before inflight replacecommit

2023-03-02 Thread via GitHub
SteNicholas commented on PR #7891: URL: https://github.com/apache/hudi/pull/7891#issuecomment-1451591769 @bvaradar, the ` timeline.containsOrBeforeTimelineStarts()` returns true for the file slice of the pending clustering instant when the previous commits before the pending clustering

[GitHub] [hudi] lokeshj1703 commented on a diff in pull request #8041: [HUDI-5847] Add support for multiple metric reporters and metric labels

2023-03-02 Thread via GitHub
lokeshj1703 commented on code in PR #8041: URL: https://github.com/apache/hudi/pull/8041#discussion_r1122837033 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/MetricUtils.java: ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [hudi] hudi-bot commented on pull request #7895: [HUDI-5736] Common de-coupling column drop flag and schema validation flag

2023-03-02 Thread via GitHub
hudi-bot commented on PR #7895: URL: https://github.com/apache/hudi/pull/7895#issuecomment-1451580764 ## CI report: * 3b6dbf0bcc7059a7dc4a5132bf45d0d1451d7fb0 UNKNOWN * c9d463794567d596503a7f9519325b88aa768f26 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7895: [HUDI-5736] Common de-coupling column drop flag and schema validation flag

2023-03-02 Thread via GitHub
hudi-bot commented on PR #7895: URL: https://github.com/apache/hudi/pull/7895#issuecomment-1451571464 ## CI report: * 3b6dbf0bcc7059a7dc4a5132bf45d0d1451d7fb0 UNKNOWN * c9d463794567d596503a7f9519325b88aa768f26 Azure:

[GitHub] [hudi] DavidZ1 commented on issue #8071: [SUPPORT]How to improve the speed of Flink writing to hudi ?

2023-03-02 Thread via GitHub
DavidZ1 commented on issue #8071: URL: https://github.com/apache/hudi/issues/8071#issuecomment-1451544322 No,We have tried the insert mode, combined with the mor and cow table formats, but the write throughput still cannot be improved. -- This is an automated message from the Apache Git

[GitHub] [hudi] hudi-bot commented on pull request #7978: [HUDI-5812] Optimize the data size check in HoodieBaseParquetWriter

2023-03-02 Thread via GitHub
hudi-bot commented on PR #7978: URL: https://github.com/apache/hudi/pull/7978#issuecomment-1451504605 ## CI report: * e4b88c67ef41c6bb7c17d2f3bd1acfd19f630132 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #7978: [HUDI-5812] Optimize the data size check in HoodieBaseParquetWriter

2023-03-02 Thread via GitHub
hudi-bot commented on PR #7978: URL: https://github.com/apache/hudi/pull/7978#issuecomment-1451495621 ## CI report: * e4b88c67ef41c6bb7c17d2f3bd1acfd19f630132 Azure: