[jira] [Commented] (HUDI-4813) Infer keygen not work in sparksql side

2022-09-15 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605625#comment-17605625 ] Danny Chen commented on HUDI-4813: -- Fixed via master branch: 3faddb7da09e5e11d1b126ba49cea4ebdeba8fc7 >

[jira] [Resolved] (HUDI-4813) Infer keygen not work in sparksql side

2022-09-15 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-4813. -- > Infer keygen not work in sparksql side > -- > > Key:

[jira] [Updated] (HUDI-4813) Infer keygen not work in sparksql side

2022-09-15 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-4813: - Fix Version/s: 0.12.1 > Infer keygen not work in sparksql side > -- >

[hudi] branch master updated: [HUDI-4813] Fix infer keygen not work in sparksql side issue (#6634)

2022-09-15 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 3faddb7da0 [HUDI-4813] Fix infer keygen not

[GitHub] [hudi] danny0405 merged pull request #6634: [HUDI-4813] Fix infer keygen not work in sparksql side issue

2022-09-15 Thread GitBox
danny0405 merged PR #6634: URL: https://github.com/apache/hudi/pull/6634 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Resolved] (HUDI-4853) Get field by name for OverwriteNonDefaultsWithLatestAvroPayload to avoid schema mismatch

2022-09-15 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-4853. -- > Get field by name for OverwriteNonDefaultsWithLatestAvroPayload to avoid > schema mismatch >

[jira] [Commented] (HUDI-4853) Get field by name for OverwriteNonDefaultsWithLatestAvroPayload to avoid schema mismatch

2022-09-15 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605624#comment-17605624 ] Danny Chen commented on HUDI-4853: -- Fixed via master branch: f70678f4354c6264b6a1e38900dd7a11cb345b96 >

[hudi] branch master updated: [HUDI-4853] Get field by name for OverwriteNonDefaultsWithLatestAvroPayload to avoid schema mismatch (#6689)

2022-09-15 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new f70678f435 [HUDI-4853] Get field by name for

[GitHub] [hudi] danny0405 merged pull request #6689: [HUDI-4853] Get field by name for OverwriteNonDefaultsWithLatestAvroP…

2022-09-15 Thread GitBox
danny0405 merged PR #6689: URL: https://github.com/apache/hudi/pull/6689 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Updated] (HUDI-4760) Clustering results in repeated triggers of clustering execution

2022-09-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4760: - Priority: Blocker (was: Major) > Clustering results in repeated triggers of clustering execution >

[jira] [Updated] (HUDI-4724) add function of skip the _rt suffix for read snapshot

2022-09-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4724: - Status: Patch Available (was: In Progress) > add function of skip the _rt suffix for read snapshot >

[jira] [Updated] (HUDI-4724) add function of skip the _rt suffix for read snapshot

2022-09-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4724: - Status: In Progress (was: Open) > add function of skip the _rt suffix for read snapshot >

[GitHub] [hudi] xushiyan commented on a diff in pull request #6537: [HUDI-4762] Avoid update metastore schema if only missing column in input

2022-09-15 Thread GitBox
xushiyan commented on code in PR #6537: URL: https://github.com/apache/hudi/pull/6537#discussion_r972634507 ## hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java: ## @@ -286,7 +286,11 @@ private boolean syncSchema(String tableName, boolean

[jira] [Updated] (HUDI-4854) Deltastreamer does not respect partition selector regex for metadata-only bootstrap

2022-09-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-4854: Status: In Progress (was: Open) > Deltastreamer does not respect partition selector regex for

[GitHub] [hudi] prasannarajaperumal commented on a diff in pull request #6476: [HUDI-3478] Support CDC for Spark in Hudi

2022-09-15 Thread GitBox
prasannarajaperumal commented on code in PR #6476: URL: https://github.com/apache/hudi/pull/6476#discussion_r972627223 ## hudi-common/src/main/java/org/apache/hudi/avro/SerializableRecord.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [hudi] prasannarajaperumal commented on a diff in pull request #6476: [HUDI-3478] Support CDC for Spark in Hudi

2022-09-15 Thread GitBox
prasannarajaperumal commented on code in PR #6476: URL: https://github.com/apache/hudi/pull/6476#discussion_r972627223 ## hudi-common/src/main/java/org/apache/hudi/avro/SerializableRecord.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [hudi] prasannarajaperumal commented on a diff in pull request #6476: [HUDI-3478] Support CDC for Spark in Hudi

2022-09-15 Thread GitBox
prasannarajaperumal commented on code in PR #6476: URL: https://github.com/apache/hudi/pull/6476#discussion_r972627223 ## hudi-common/src/main/java/org/apache/hudi/avro/SerializableRecord.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [hudi] Zhangshunyu commented on issue #6691: [SUPPORT]Error after applyed HUDI-4851 for InSet

2022-09-15 Thread GitBox
Zhangshunyu commented on issue #6691: URL: https://github.com/apache/hudi/issues/6691#issuecomment-1248933132 select count(*),count(distinct word) from test_table where word in ('HelloWorld', 'OK', ... etc. 1000 words here)

[GitHub] [hudi] Zhangshunyu opened a new issue, #6691: [SUPPORT]Error while applyed HUDI-4851 for InSet

2022-09-15 Thread GitBox
Zhangshunyu opened a new issue, #6691: URL: https://github.com/apache/hudi/issues/6691 ``` Caused by: java.lang.RuntimeException: Unsupported literal type class org.apache.spark.unsafe.types.UTF8String HelloWorld at

[hudi] branch asf-site updated: fix: blog image landing page (#6690)

2022-09-15 Thread bhavanisudha
This is an automated email from the ASF dual-hosted git repository. bhavanisudha pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 3b655aa086 fix: blog image landing page

[GitHub] [hudi] bhasudha merged pull request #6690: [DOCS] fix: blog image landing page

2022-09-15 Thread GitBox
bhasudha merged PR #6690: URL: https://github.com/apache/hudi/pull/6690 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Updated] (HUDI-4762) Hive sync update schema removes columns

2022-09-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4762: - Status: In Progress (was: Open) > Hive sync update schema removes columns >

[jira] [Updated] (HUDI-4762) Hive sync update schema removes columns

2022-09-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-4762: - Status: Patch Available (was: In Progress) > Hive sync update schema removes columns >

[jira] [Closed] (HUDI-3861) 'path' in CatalogTable#properties failed to be updated when renaming table

2022-09-15 Thread Raymond Xu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu closed HUDI-3861. Resolution: Fixed > 'path' in CatalogTable#properties failed to be updated when renaming table >

[hudi] branch master updated (bf64e60d31 -> c2b72306bd)

2022-09-15 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from bf64e60d31 [HUDI-4796] MetricsReporter stop bug (#6619) add c2b72306bd [HUDI-3861] update tblp 'path' when

[GitHub] [hudi] xushiyan merged pull request #5320: [HUDI-3861] update tblp 'path' when rename table

2022-09-15 Thread GitBox
xushiyan merged PR #5320: URL: https://github.com/apache/hudi/pull/5320 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] pintusoliya opened a new pull request, #6690: fix: blog image landing page

2022-09-15 Thread GitBox
pintusoliya opened a new pull request, #6690: URL: https://github.com/apache/hudi/pull/6690 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any

[GitHub] [hudi] hudi-bot commented on pull request #6689: [HUDI-4853] Get field by name for OverwriteNonDefaultsWithLatestAvroP…

2022-09-15 Thread GitBox
hudi-bot commented on PR #6689: URL: https://github.com/apache/hudi/pull/6689#issuecomment-1248921800 ## CI report: * cab4b6a3b31aff9a0aa4a825d341346aaa7ede73 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #6358: [HUDI-4588][HUDI-4472] Fixing `HoodieParquetReader` to properly specify projected schema when reading Parquet file

2022-09-15 Thread GitBox
hudi-bot commented on PR #6358: URL: https://github.com/apache/hudi/pull/6358#issuecomment-1248921478 ## CI report: * 288d166c49602a4593b1e97763a467811903737d UNKNOWN * c4b6bb8dc7a4ddce5f729e5a49ac10aad25e8931 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #3985: [HUDI-2754] Performance improvement for IncrementalRelation

2022-09-15 Thread GitBox
hudi-bot commented on PR #3985: URL: https://github.com/apache/hudi/pull/3985#issuecomment-1248920688 ## CI report: * ccd1d89352a2f72feb381962718cc0c80920c041 UNKNOWN * dee3b8154dfc173ea70352986a7ebdd028a968b0 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #6676: [HUDI-4453] Fix schema to include partition columns in bootstrap operation

2022-09-15 Thread GitBox
hudi-bot commented on PR #6676: URL: https://github.com/apache/hudi/pull/6676#issuecomment-1248919461 ## CI report: * fa203bff2e2bb9fc27e50f0b0c2613770bfa5dc6 Azure:

[GitHub] [hudi] codope commented on a diff in pull request #6548: [HUDI-4749] Fixing full cleaning to leverage metadata table

2022-09-15 Thread GitBox
codope commented on code in PR #6548: URL: https://github.com/apache/hudi/pull/6548#discussion_r972613010 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java: ## @@ -206,15 +206,7 @@ private List

[GitHub] [hudi] hudi-bot commented on pull request #6358: [HUDI-4588][HUDI-4472] Fixing `HoodieParquetReader` to properly specify projected schema when reading Parquet file

2022-09-15 Thread GitBox
hudi-bot commented on PR #6358: URL: https://github.com/apache/hudi/pull/6358#issuecomment-1248919216 ## CI report: * 288d166c49602a4593b1e97763a467811903737d UNKNOWN * c4b6bb8dc7a4ddce5f729e5a49ac10aad25e8931 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #3985: [HUDI-2754] Performance improvement for IncrementalRelation

2022-09-15 Thread GitBox
hudi-bot commented on PR #3985: URL: https://github.com/apache/hudi/pull/3985#issuecomment-1248918482 ## CI report: * ccd1d89352a2f72feb381962718cc0c80920c041 UNKNOWN * dee3b8154dfc173ea70352986a7ebdd028a968b0 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #6688: Fix AWSDmsAvroPayload#combineAndGetUpdateValue when using MOR snapshot query after delete operations

2022-09-15 Thread GitBox
hudi-bot commented on PR #6688: URL: https://github.com/apache/hudi/pull/6688#issuecomment-1248917319 ## CI report: * fff1405467fb5f6a7fdb6d3d043714e268f1c875 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #6670: [HUDI-4842] Support compression strategy based on delte file length

2022-09-15 Thread GitBox
hudi-bot commented on PR #6670: URL: https://github.com/apache/hudi/pull/6670#issuecomment-1248891059 ## CI report: * 462f77736f855dc277cc62e0778fb4c1fa04f09a Azure:

[GitHub] [hudi] hudi-bot commented on pull request #6670: [HUDI-4842] Support compression strategy based on delte file length

2022-09-15 Thread GitBox
hudi-bot commented on PR #6670: URL: https://github.com/apache/hudi/pull/6670#issuecomment-1248889062 ## CI report: * 462f77736f855dc277cc62e0778fb4c1fa04f09a Azure:

[GitHub] [hudi] hudi-bot commented on pull request #6677: [HUDI-4294][Stacked on 4293] Introduce build action to actually perform index data generation

2022-09-15 Thread GitBox
hudi-bot commented on PR #6677: URL: https://github.com/apache/hudi/pull/6677#issuecomment-1248887133 ## CI report: * 0ce0aee73e1641f071abdfc44d4f5473a425befb Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5933: [HUDI-4293] Implement Create/Drop/Show/Refresh Index Command for Secondary Index

2022-09-15 Thread GitBox
hudi-bot commented on PR #5933: URL: https://github.com/apache/hudi/pull/5933#issuecomment-1248886699 ## CI report: * 65359879df848d75b6693f4c313dc9453d635edd Azure:

[GitHub] [hudi] xushiyan commented on a diff in pull request #6662: [HUDI-4832] Fix drop partition meta sync

2022-09-15 Thread GitBox
xushiyan commented on code in PR #6662: URL: https://github.com/apache/hudi/pull/6662#discussion_r972587570 ## hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/HoodieSyncClient.java: ## @@ -158,4 +171,23 @@ public List getPartitionEvents(List

[GitHub] [hudi] hudi-bot commented on pull request #6677: [HUDI-4294][Stacked on 4293] Introduce build action to actually perform index data generation

2022-09-15 Thread GitBox
hudi-bot commented on PR #6677: URL: https://github.com/apache/hudi/pull/6677#issuecomment-1248884818 ## CI report: * 0ce0aee73e1641f071abdfc44d4f5473a425befb Azure:

[GitHub] [hudi] hudi-bot commented on pull request #5933: [HUDI-4293] Implement Create/Drop/Show/Refresh Index Command for Secondary Index

2022-09-15 Thread GitBox
hudi-bot commented on PR #5933: URL: https://github.com/apache/hudi/pull/5933#issuecomment-1248884303 ## CI report: * 65359879df848d75b6693f4c313dc9453d635edd Azure:

[GitHub] [hudi] TJX2014 commented on pull request #6630: [HUDI-4808] Fix HoodieSimpleBucketIndex not consider bucket num in lo…

2022-09-15 Thread GitBox
TJX2014 commented on PR #6630: URL: https://github.com/apache/hudi/pull/6630#issuecomment-1248882064 Hi @danny0405 ci succeed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] TJX2014 commented on pull request #6634: [HUDI-4813] Fix infer keygen not work in sparksql side issue

2022-09-15 Thread GitBox
TJX2014 commented on PR #6634: URL: https://github.com/apache/hudi/pull/6634#issuecomment-1248881741 Hi @danny0405 ci passed : ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hudi] nsivabalan commented on issue #6611: SchemaEvolution : Default value not getting fetched properly for not null columns from confluent kafka schema registry

2022-09-15 Thread GitBox
nsivabalan commented on issue #6611: URL: https://github.com/apache/hudi/issues/6611#issuecomment-1248879098 I am yet to try this out locally and see how this pans out.bcoz, you have custom default value for strings. usually null defaults are taken into consideration. but non null

[GitHub] [hudi] nsivabalan commented on issue #5249: [SUPPORT] Deltastreamer job does not terminate on Kubernetes when hoodie.metrics.on=true

2022-09-15 Thread GitBox
nsivabalan commented on issue #5249: URL: https://github.com/apache/hudi/issues/5249#issuecomment-1248874327  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hudi] nsivabalan commented on issue #6606: Observing data duplication with Single Writer

2022-09-15 Thread GitBox
nsivabalan commented on issue #6606: URL: https://github.com/apache/hudi/issues/6606#issuecomment-1248873001 you can read about multi writer guarantees here https://hudi.apache.org/docs/concurrency_control#multi-writer-guarantees -- This is an automated message from the Apache Git

[GitHub] [hudi] nsivabalan commented on issue #6606: Observing data duplication with Single Writer

2022-09-15 Thread GitBox
nsivabalan commented on issue #6606: URL: https://github.com/apache/hudi/issues/6606#issuecomment-1248872759 here is what is happening. if there are two concurrent writers writing to non overlapping data files, hudi will succeed both writes. but if both are modifying the same data file,

[GitHub] [hudi] danny0405 commented on a diff in pull request #4676: [HUDI-3304] Support partial update payload

2022-09-15 Thread GitBox
danny0405 commented on code in PR #4676: URL: https://github.com/apache/hudi/pull/4676#discussion_r972565604 ## hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteNonDefaultsWithLatestAvroPayload.java: ## @@ -58,19 +58,32 @@ public Option

[GitHub] [hudi] scxwhite commented on a diff in pull request #6670: [HUDI-4842] Support compression strategy based on delte file length

2022-09-15 Thread GitBox
scxwhite commented on code in PR #6670: URL: https://github.com/apache/hudi/pull/6670#discussion_r972559146 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java: ## @@ -106,6 +106,12 @@ public class HoodieCompactionConfig extends

[GitHub] [hudi] hudi-bot commented on pull request #6489: [HUDI-4485] [cli] Bumped spring shell to 2.1.1. Updated the default …

2022-09-15 Thread GitBox
hudi-bot commented on PR #6489: URL: https://github.com/apache/hudi/pull/6489#issuecomment-1248845860 ## CI report: * 7ea1f728918e22be5e545f0b565f4321f2e43143 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #6358: [HUDI-4588][HUDI-4472] Fixing `HoodieParquetReader` to properly specify projected schema when reading Parquet file

2022-09-15 Thread GitBox
hudi-bot commented on PR #6358: URL: https://github.com/apache/hudi/pull/6358#issuecomment-1248845763 ## CI report: * 288d166c49602a4593b1e97763a467811903737d UNKNOWN * c4b6bb8dc7a4ddce5f729e5a49ac10aad25e8931 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #4676: [HUDI-3304] Support partial update payload

2022-09-15 Thread GitBox
hudi-bot commented on PR #4676: URL: https://github.com/apache/hudi/pull/4676#issuecomment-1248844971 ## CI report: * 5944f5cbe9ce73fe6b7e27a0d381eaeb80dead38 UNKNOWN * 4ef7b451c3dd795906f3f68571256baeb330a59f UNKNOWN * 6aeb3d0d8f09aeab2a5766cf9d25ecb30537 UNKNOWN *

[GitHub] [hudi] scxwhite commented on a diff in pull request #6670: [HUDI-4842] Support compression strategy based on delte file length

2022-09-15 Thread GitBox
scxwhite commented on code in PR #6670: URL: https://github.com/apache/hudi/pull/6670#discussion_r972557765 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java: ## @@ -106,6 +106,12 @@ public class HoodieCompactionConfig extends

[GitHub] [hudi] xushiyan commented on a diff in pull request #6652: [HUDI-4830] Fix testNoGlobalConfFileConfigured when add hudi-defaults.conf in default dir

2022-09-15 Thread GitBox
xushiyan commented on code in PR #6652: URL: https://github.com/apache/hudi/pull/6652#discussion_r972557608 ## hudi-common/src/test/java/org/apache/hudi/common/util/TestDFSPropertiesConfiguration.java: ## @@ -173,7 +173,9 @@ public void testNoGlobalConfFileConfigured() {

[GitHub] [hudi] xushiyan commented on a diff in pull request #6652: [HUDI-4830] Fix testNoGlobalConfFileConfigured when add hudi-defaults.conf in default dir

2022-09-15 Thread GitBox
xushiyan commented on code in PR #6652: URL: https://github.com/apache/hudi/pull/6652#discussion_r972557066 ## hudi-common/src/test/java/org/apache/hudi/common/util/TestDFSPropertiesConfiguration.java: ## @@ -173,7 +173,9 @@ public void testNoGlobalConfFileConfigured() {

[GitHub] [hudi] hudi-bot commented on pull request #6489: [HUDI-4485] [cli] Bumped spring shell to 2.1.1. Updated the default …

2022-09-15 Thread GitBox
hudi-bot commented on PR #6489: URL: https://github.com/apache/hudi/pull/6489#issuecomment-1248843297 ## CI report: * 7ea1f728918e22be5e545f0b565f4321f2e43143 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #6358: [HUDI-4588][HUDI-4472] Fixing `HoodieParquetReader` to properly specify projected schema when reading Parquet file

2022-09-15 Thread GitBox
hudi-bot commented on PR #6358: URL: https://github.com/apache/hudi/pull/6358#issuecomment-1248843191 ## CI report: * 288d166c49602a4593b1e97763a467811903737d UNKNOWN * c4b6bb8dc7a4ddce5f729e5a49ac10aad25e8931 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #4676: [HUDI-3304] Support partial update payload

2022-09-15 Thread GitBox
hudi-bot commented on PR #4676: URL: https://github.com/apache/hudi/pull/4676#issuecomment-1248842484 ## CI report: * 5944f5cbe9ce73fe6b7e27a0d381eaeb80dead38 UNKNOWN * 4ef7b451c3dd795906f3f68571256baeb330a59f UNKNOWN * 6aeb3d0d8f09aeab2a5766cf9d25ecb30537 UNKNOWN *

[GitHub] [hudi] hudi-bot commented on pull request #6676: [HUDI-4453] Fix schema to include partition columns in bootstrap operation

2022-09-15 Thread GitBox
hudi-bot commented on PR #6676: URL: https://github.com/apache/hudi/pull/6676#issuecomment-1248840961 ## CI report: * fa203bff2e2bb9fc27e50f0b0c2613770bfa5dc6 Azure:

[jira] [Updated] (HUDI-4855) Bootstrap table from Deltastreamer cannot be read in Spark

2022-09-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-4855: Sprint: 2022/09/05 > Bootstrap table from Deltastreamer cannot be read in Spark >

[jira] [Updated] (HUDI-4854) Deltastreamer does not respect partition selector regex for metadata-only bootstrap

2022-09-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-4854: Sprint: 2022/09/05 > Deltastreamer does not respect partition selector regex for metadata-only > bootstrap

[GitHub] [hudi] paul8263 commented on pull request #6489: [HUDI-4485] [cli] Bumped spring shell to 2.1.1. Updated the default …

2022-09-15 Thread GitBox
paul8263 commented on PR #6489: URL: https://github.com/apache/hudi/pull/6489#issuecomment-1248831904 Pushed to solve the conflicts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[jira] [Updated] (HUDI-4855) Bootstrap table from Deltastreamer cannot be read in Spark

2022-09-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-4855: Component/s: bootstrap > Bootstrap table from Deltastreamer cannot be read in Spark >

[jira] [Updated] (HUDI-4855) Bootstrap table from Deltastreamer cannot be read in Spark

2022-09-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-4855: Description:   {code:java} scala> val df = spark.read.format("hudi").load("")

[jira] [Updated] (HUDI-4855) Bootstrap table from Deltastreamer cannot be read in Spark

2022-09-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-4855: Epic Link: HUDI-1265 Story Points: 1 > Bootstrap table from Deltastreamer cannot be read in Spark >

[jira] [Updated] (HUDI-4855) Bootstrap table from Deltastreamer cannot be read in Spark

2022-09-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-4855: Fix Version/s: 0.12.1 > Bootstrap table from Deltastreamer cannot be read in Spark >

[jira] [Created] (HUDI-4855) Bootstrap table from Deltastreamer cannot be read in Spark

2022-09-15 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-4855: --- Summary: Bootstrap table from Deltastreamer cannot be read in Spark Key: HUDI-4855 URL: https://issues.apache.org/jira/browse/HUDI-4855 Project: Apache Hudi Issue

[jira] [Updated] (HUDI-4854) Deltastreamer does not respect partition selector regex for bootstrap

2022-09-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-4854: Fix Version/s: 0.12.1 > Deltastreamer does not respect partition selector regex for bootstrap >

[jira] [Updated] (HUDI-4854) Deltastreamer does not respect partition selector regex for metadata-only bootstrap

2022-09-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-4854: Summary: Deltastreamer does not respect partition selector regex for metadata-only bootstrap (was:

[jira] [Updated] (HUDI-4854) Deltastreamer does not respect partition selector regex for bootstrap

2022-09-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-4854: Story Points: 1 Issue Type: Bug (was: Improvement) > Deltastreamer does not respect partition

[jira] [Updated] (HUDI-4854) Deltastreamer does not respect partition selector regex for bootstrap

2022-09-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-4854: Component/s: bootstrap Epic Link: HUDI-1265 > Deltastreamer does not respect partition selector regex

[jira] [Assigned] (HUDI-4854) Deltastreamer does not respect partition selector regex for bootstrap

2022-09-15 Thread Ethan Guo (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-4854: --- Assignee: Ethan Guo > Deltastreamer does not respect partition selector regex for bootstrap >

[jira] [Created] (HUDI-4854) Deltastreamer does not respect partition selector regex for bootstrap

2022-09-15 Thread Ethan Guo (Jira)
Ethan Guo created HUDI-4854: --- Summary: Deltastreamer does not respect partition selector regex for bootstrap Key: HUDI-4854 URL: https://issues.apache.org/jira/browse/HUDI-4854 Project: Apache Hudi

[hudi] branch master updated: [HUDI-4796] MetricsReporter stop bug (#6619)

2022-09-15 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new bf64e60d31 [HUDI-4796] MetricsReporter stop bug

[GitHub] [hudi] yihua merged pull request #6619: [HUDI-4796] MetricsReporter stop bug

2022-09-15 Thread GitBox
yihua merged PR #6619: URL: https://github.com/apache/hudi/pull/6619 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] yihua closed issue #6517: [SUPPORT] when using bootstrap partitioned table, partition column return null when select table

2022-09-15 Thread GitBox
yihua closed issue #6517: [SUPPORT] when using bootstrap partitioned table, partition column return null when select table URL: https://github.com/apache/hudi/issues/6517 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [hudi] yihua commented on issue #6517: [SUPPORT] when using bootstrap partitioned table, partition column return null when select table

2022-09-15 Thread GitBox
yihua commented on issue #6517: URL: https://github.com/apache/hudi/issues/6517#issuecomment-1248823659 #6673 and #6676 have fixed the problem of reading the partition column from a bootstrap table and I verified that it works (see the `df.show` result below after bootstrap). Closing this

[hudi] branch master updated (22d6019559 -> 6e31b7cef4)

2022-09-15 Thread xushiyan
This is an automated email from the ASF dual-hosted git repository. xushiyan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 22d6019559 [HUDI-4706] Fix InternalSchemaChangeApplier#applyAddChange error to add nest type (#6486) add

[GitHub] [hudi] yihua commented on a diff in pull request #6670: [HUDI-4842] Support compression strategy based on delte file length

2022-09-15 Thread GitBox
yihua commented on code in PR #6670: URL: https://github.com/apache/hudi/pull/6670#discussion_r972540738 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java: ## @@ -106,6 +106,12 @@ public class HoodieCompactionConfig extends

[GitHub] [hudi] xushiyan closed issue #6655: [SUPPORT] tryComposeIndexFilterExpr in dataskip util could support InSet expression of spark?

2022-09-15 Thread GitBox
xushiyan closed issue #6655: [SUPPORT] tryComposeIndexFilterExpr in dataskip util could support InSet expression of spark? URL: https://github.com/apache/hudi/issues/6655 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [hudi] xushiyan merged pull request #6685: [HUDI-4851] Fixing CSI not handling `InSet` operator properly

2022-09-15 Thread GitBox
xushiyan merged PR #6685: URL: https://github.com/apache/hudi/pull/6685 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[hudi] branch master updated (488f58d770 -> 22d6019559)

2022-09-15 Thread mengtao
This is an automated email from the ASF dual-hosted git repository. mengtao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 488f58d770 [HUDI-4785] Fix partition discovery in bootstrap operation (#6673) add 22d6019559 [HUDI-4706] Fix

[GitHub] [hudi] xiarixiaoyao merged pull request #6486: [HUDI-4706] Fix InternalSchemaChangeApplier#applyAddChange error to add nest type

2022-09-15 Thread GitBox
xiarixiaoyao merged PR #6486: URL: https://github.com/apache/hudi/pull/6486 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] yihua commented on a diff in pull request #6670: [HUDI-4842] Support compression strategy based on delte file length

2022-09-15 Thread GitBox
yihua commented on code in PR #6670: URL: https://github.com/apache/hudi/pull/6670#discussion_r972532363 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java: ## @@ -241,41 +255,61 @@ public class HoodieCompactionConfig extends

[GitHub] [hudi] hudi-bot commented on pull request #6676: [HUDI-4453] Fix schema to include partition columns in bootstrap operation

2022-09-15 Thread GitBox
hudi-bot commented on PR #6676: URL: https://github.com/apache/hudi/pull/6676#issuecomment-1248813922 ## CI report: * fa203bff2e2bb9fc27e50f0b0c2613770bfa5dc6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #6689: [HUDI-4853] Get field by name for OverwriteNonDefaultsWithLatestAvroP…

2022-09-15 Thread GitBox
hudi-bot commented on PR #6689: URL: https://github.com/apache/hudi/pull/6689#issuecomment-1248808740 ## CI report: * cab4b6a3b31aff9a0aa4a825d341346aaa7ede73 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #6688: Fix AWSDmsAvroPayload#combineAndGetUpdateValue when using MOR snapshot query after delete operations

2022-09-15 Thread GitBox
hudi-bot commented on PR #6688: URL: https://github.com/apache/hudi/pull/6688#issuecomment-1248808723 ## CI report: * fff1405467fb5f6a7fdb6d3d043714e268f1c875 Azure:

[GitHub] [hudi] hudi-bot commented on pull request #6689: [HUDI-4853] Get field by name for OverwriteNonDefaultsWithLatestAvroP…

2022-09-15 Thread GitBox
hudi-bot commented on PR #6689: URL: https://github.com/apache/hudi/pull/6689#issuecomment-1248806292 ## CI report: * cab4b6a3b31aff9a0aa4a825d341346aaa7ede73 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #6688: Fix AWSDmsAvroPayload#combineAndGetUpdateValue when using MOR snapshot query after delete operations

2022-09-15 Thread GitBox
hudi-bot commented on PR #6688: URL: https://github.com/apache/hudi/pull/6688#issuecomment-1248806264 ## CI report: * fff1405467fb5f6a7fdb6d3d043714e268f1c875 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the

[GitHub] [hudi] hudi-bot commented on pull request #6615: [HUDI-4758] Add validations to java spark examples

2022-09-15 Thread GitBox
hudi-bot commented on PR #6615: URL: https://github.com/apache/hudi/pull/6615#issuecomment-1248803426 ## CI report: * d675d338c90b09abbdbcc84003873cf05c40f871 Azure:

[hudi] branch master updated: [HUDI-4785] Fix partition discovery in bootstrap operation (#6673)

2022-09-15 Thread yihua
This is an automated email from the ASF dual-hosted git repository. yihua pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 488f58d770 [HUDI-4785] Fix partition discovery in

[GitHub] [hudi] yihua merged pull request #6673: [HUDI-4785] Fix partition discovery in bootstrap operation

2022-09-15 Thread GitBox
yihua merged PR #6673: URL: https://github.com/apache/hudi/pull/6673 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hudi] yihua commented on pull request #6673: [HUDI-4785] Fix partition discovery in bootstrap operation

2022-09-15 Thread GitBox
yihua commented on PR #6673: URL: https://github.com/apache/hudi/pull/6673#issuecomment-1248801714 Merging this as the rebasing only touches the `TestDataSourceForBootstrap` and it passes locally. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] 5herhom commented on pull request #6031: [HUDI-4282] Repair IOException in some other dfs, except hdfs,when check block corrupted in HoodieLogFileReader

2022-09-15 Thread GitBox
5herhom commented on PR #6031: URL: https://github.com/apache/hudi/pull/6031#issuecomment-1248796155 > @5herhom : can you follow up on the feedback. its nearing landing. Sorry, I'm busy these days. I will commit in two days -- This is an automated message from the Apache Git

[jira] [Updated] (HUDI-4853) Get field by name for OverwriteNonDefaultsWithLatestAvroPayload to avoid schema mismatch

2022-09-15 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-4853: - Labels: pull-request-available (was: ) > Get field by name for

[GitHub] [hudi] danny0405 opened a new pull request, #6689: [HUDI-4853] Get field by name for OverwriteNonDefaultsWithLatestAvroP…

2022-09-15 Thread GitBox
danny0405 opened a new pull request, #6689: URL: https://github.com/apache/hudi/pull/6689 …ayload to avoid schema mismatch ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or

[GitHub] [hudi] paul8263 commented on a diff in pull request #6489: [HUDI-4485] [cli] Bumped spring shell to 2.1.1. Updated the default …

2022-09-15 Thread GitBox
paul8263 commented on code in PR #6489: URL: https://github.com/apache/hudi/pull/6489#discussion_r972521635 ## hudi-cli/src/main/java/org/apache/hudi/cli/commands/SparkMain.java: ## @@ -86,7 +87,7 @@ */ public class SparkMain { - private static final Logger LOG =

[GitHub] [hudi] rahil-c opened a new pull request, #6688: Fix AWSDmsAvroPayload#combineAndGetUpdateValue when using MOR snapshot query after delete operations

2022-09-15 Thread GitBox
rahil-c opened a new pull request, #6688: URL: https://github.com/apache/hudi/pull/6688 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any performance

[GitHub] [hudi] yihua commented on a diff in pull request #6673: [HUDI-4785] Fix partition discovery in bootstrap operation

2022-09-15 Thread GitBox
yihua commented on code in PR #6673: URL: https://github.com/apache/hudi/pull/6673#discussion_r972517320 ## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBootstrapRelation.scala: ## @@ -147,7 +146,7 @@ class HoodieBootstrapRelation(@transient val

  1   2   3   4   >