[jira] [Resolved] (HUDI-2204) Add marker files for flink writer

2021-07-21 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen resolved HUDI-2204. -- Resolution: Fixed Fixed via master branch: 2370a9facbe4418f994f29c426e9b2a255e3abb0 > Add marker files

[jira] [Commented] (HUDI-2204) Add marker files for flink writer

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385266#comment-17385266 ] ASF GitHub Bot commented on HUDI-2204: -- danny0405 merged pull request #3316: URL:

[hudi] branch master updated (5a94b6b -> 2370a9f)

2021-07-21 Thread danny0405
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 5a94b6b [HUDI-2192] Clean up Multiple versions of scala libraries detected Warning (#3292) add 2370a9f

[GitHub] [hudi] danny0405 merged pull request #3316: [HUDI-2204] Add marker files for flink writer

2021-07-21 Thread GitBox
danny0405 merged pull request #3316: URL: https://github.com/apache/hudi/pull/3316 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan

2021-07-21 Thread ASF GitHub Bot (Jira)
.clustering.plan.strategy.daybased.lookback.partitions` to  control > the number of partitions to list from the latest partition to create > ClusteringPlan. > For example, we have 6 partitions based on date, and users set  > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 &

[GitHub] [hudi] zhangyue19921010 commented on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan

2021-07-21 Thread GitBox
zhangyue19921010 commented on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-884662878 Hi @satishkotha Thanks for your review. > @zhangyue19921010 LGTM. Is it possible to add a test? Sure thing, added. PTAL :) -- This is an automated

[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan

2021-07-21 Thread ASF GitHub Bot (Jira)
> ClusteringPlan. > For example, we have 6 partitions based on date, and users set  > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| >                                                          

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan

2021-07-21 Thread GitBox
codecov-commenter edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882457453 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan

2021-07-21 Thread ASF GitHub Bot (Jira)
o  control > the number of partitions to list from the latest partition to create > ClusteringPlan. > For example, we have 6 partitions based on date, and users set  > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 >

[GitHub] [hudi] hudi-bot edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882429208 ## CI report: * 4b85d50e5c18695641d1c91edccf6a638260761a Azure:

[GitHub] [hudi] vinothchandar closed issue #3236: why use avro not parquet in MOR

2021-07-21 Thread GitBox
vinothchandar closed issue #3236: URL: https://github.com/apache/hudi/issues/3236 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan

2021-07-21 Thread ASF GitHub Bot (Jira)
umber of partitions to list from the latest partition to create > ClusteringPlan. > For example, we have 6 partitions based on date, and users set  > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| >                                                    

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan

2021-07-21 Thread GitBox
codecov-commenter edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882457453 #

[GitHub] [hudi] hudi-bot edited a comment on pull request #3325: [WIP] Fixing payload instantiation to include preCombine field in LogRecordScanner

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3325: URL: https://github.com/apache/hudi/pull/3325#issuecomment-884643521 ## CI report: * 7ab74d46bc22b7e93e0ac00a2200b62c4bd120e6 Azure:

[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan

2021-07-21 Thread ASF GitHub Bot (Jira)
ple, we have 6 partitions based on date, and users set  > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| >                                                                               >   |<- c

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan

2021-07-21 Thread GitBox
codecov-commenter edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882457453 #

[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385251#comment-17385251 ] ASF GitHub Bot commented on HUDI-1771: -- hudi-bot edited a comment on pull request #3285: URL:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * a46c6f21415c65e005ba9ac267cd2bda4528a39b Azure:

[GitHub] [hudi] nsivabalan commented on issue #3313: [SUPPORT] CoW: Hudi Upsert not working when there is a timestamp field in the composite key

2021-07-21 Thread GitBox
nsivabalan commented on issue #3313: URL: https://github.com/apache/hudi/issues/3313#issuecomment-884649681 Would you mind giving me steps to reproduce w/ some sample data. guess handful of records should be good enough to reproduce. -- This is an automated message from the Apache Git

[GitHub] [hudi] nsivabalan commented on issue #3313: [SUPPORT] CoW: Hudi Upsert not working when there is a timestamp field in the composite key

2021-07-21 Thread GitBox
nsivabalan commented on issue #3313: URL: https://github.com/apache/hudi/issues/3313#issuecomment-884649224 AFAIK, CustomKeyGenerator does pick field values as is. don't think it does any timestamp related conversion. Can you confirm that during bulk_insert the claim_subm_dt field

[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan

2021-07-21 Thread ASF GitHub Bot (Jira)
to  control > the number of partitions to list from the latest partition to create > ClusteringPlan. > For example, we have 6 partitions based on date, and users set  > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 > |20210718|20210719 |20210720 |20210721 |20210722 

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan

2021-07-21 Thread GitBox
codecov-commenter edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882457453 #

[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan

2021-07-21 Thread ASF GitHub Bot (Jira)
o  control > the number of partitions to list from the latest partition to create > ClusteringPlan. > For example, we have 6 partitions based on date, and users set  > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 >

[GitHub] [hudi] hudi-bot edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882429208 ## CI report: * 350f4266dd6951cb64ba492553e1099978f23fcf Azure:

[jira] [Commented] (HUDI-2204) Add marker files for flink writer

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385244#comment-17385244 ] ASF GitHub Bot commented on HUDI-2204: -- yuzhaojing commented on pull request #3316: URL:

[GitHub] [hudi] yuzhaojing commented on pull request #3316: [HUDI-2204] Add marker files for flink writer

2021-07-21 Thread GitBox
yuzhaojing commented on pull request #3316: URL: https://github.com/apache/hudi/pull/3316#issuecomment-884645405 +1 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig

2021-07-21 Thread GitBox
codecov-commenter edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883347621 #

[jira] [Commented] (HUDI-2194) Skip the latest N partitions when creating ClusteringPlan

2021-07-21 Thread ASF GitHub Bot (Jira)
r example, we have 6 partitions based on date, and users set  > `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 > |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)| >                                                                               >   |<- cho

[GitHub] [hudi] hudi-bot edited a comment on pull request #3300: [HUDI-2194] Skip the latest N partitions when choosing partitions to create ClusteringPlan

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3300: URL: https://github.com/apache/hudi/pull/3300#issuecomment-882429208 ## CI report: * 350f4266dd6951cb64ba492553e1099978f23fcf Azure:

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig

2021-07-21 Thread GitBox
codecov-commenter edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883347621 #

[hudi] branch asf-site updated: Travis CI build asf-site

2021-07-21 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 09606d3 Travis CI build asf-site 09606d3 is

[GitHub] [hudi] hudi-bot edited a comment on pull request #3325: [WIP] Fixing payload instantiation to include preCombine field in LogRecordScanner

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3325: URL: https://github.com/apache/hudi/pull/3325#issuecomment-884643521 ## CI report: * 7ab74d46bc22b7e93e0ac00a2200b62c4bd120e6 Azure:

[jira] [Commented] (HUDI-1548) Fix documentation around schema evolution

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385241#comment-17385241 ] ASF GitHub Bot commented on HUDI-1548: -- nsivabalan merged pull request #3257: URL:

[GitHub] [hudi] nsivabalan merged pull request #3257: [HUDI-1548] Add documentation for schema evolution

2021-07-21 Thread GitBox
nsivabalan merged pull request #3257: URL: https://github.com/apache/hudi/pull/3257 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[hudi] branch asf-site updated: [HUDI-1548] Add documentation for schema evolution (#3257)

2021-07-21 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new ec8bde1 [HUDI-1548] Add documentation for

[GitHub] [hudi] hudi-bot commented on pull request #3325: [WIP] Fixing payload instantiation to include preCombine field in LogRecordScanner

2021-07-21 Thread GitBox
hudi-bot commented on pull request #3325: URL: https://github.com/apache/hudi/pull/3325#issuecomment-884643521 ## CI report: * 7ab74d46bc22b7e93e0ac00a2200b62c4bd120e6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis`

[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries

2021-07-21 Thread GitBox
nsivabalan commented on issue #3323: URL: https://github.com/apache/hudi/issues/3323#issuecomment-884642954 This fix worked for me locally: https://github.com/apache/hudi/pull/3325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [hudi] nsivabalan opened a new pull request #3325: [WIP] Fixing payload instantiation to include preCombine field in LogRecordScanner

2021-07-21 Thread GitBox
nsivabalan opened a new pull request #3325: URL: https://github.com/apache/hudi/pull/3325 …Scanner ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is

[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries

2021-07-21 Thread GitBox
nsivabalan commented on issue #3323: URL: https://github.com/apache/hudi/issues/3323#issuecomment-884640235 @vinothchandar : Do you know the reason why it was designed this way? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[jira] [Commented] (HUDI-2187) Hive integration Improvment

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385238#comment-17385238 ] ASF GitHub Bot commented on HUDI-2187: -- hudi-bot edited a comment on pull request #3289: URL:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3289: [HUDI-2187] Add a shim layer to support multiple hive version

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 602b6cdf97b686257bfb17a05d456f9d52f4c147 Azure:

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig

2021-07-21 Thread GitBox
codecov-commenter edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883347621 #

[jira] [Commented] (HUDI-2187) Hive integration Improvment

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385236#comment-17385236 ] ASF GitHub Bot commented on HUDI-2187: -- hudi-bot edited a comment on pull request #3289: URL:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3289: [HUDI-2187] Add a shim layer to support multiple hive version

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 1a9c6e49be153e7cf9646da93c1114ae75969c22 Azure:

[GitHub] [hudi] nsivabalan edited a comment on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries

2021-07-21 Thread GitBox
nsivabalan edited a comment on issue #3323: URL: https://github.com/apache/hudi/issues/3323#issuecomment-884638185 I delved deeper and found the root cause. Could be a bug in code. But surprised how come we have not encountered this so far. When we construct the records back from disk

[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries

2021-07-21 Thread GitBox
nsivabalan commented on issue #3323: URL: https://github.com/apache/hudi/issues/3323#issuecomment-884638185 I delved deeper and found the root cause. Could be a bug in code. But surprised how come we have not encountered this so far. When we construct the records back from disk (log

[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385235#comment-17385235 ] ASF GitHub Bot commented on HUDI-1771: -- hudi-bot edited a comment on pull request #3285: URL:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 4e731ff162b777b85e6c6c09e5387a2f0215197b Azure:

[jira] [Assigned] (HUDI-2188) Improve test for the insert_overwrite and insert_overwrite_table in hoodieDeltaStreamer

2021-07-21 Thread Samrat Deb (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samrat Deb reassigned HUDI-2188: Assignee: Samrat Deb > Improve test for the insert_overwrite and insert_overwrite_table in >

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig

2021-07-21 Thread GitBox
codecov-commenter edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883347621 #

[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385233#comment-17385233 ] ASF GitHub Bot commented on HUDI-1771: -- hudi-bot edited a comment on pull request #3285: URL:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 4e731ff162b777b85e6c6c09e5387a2f0215197b Azure:

[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385230#comment-17385230 ] ASF GitHub Bot commented on HUDI-1771: -- hudi-bot edited a comment on pull request #3285: URL:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 823eea932ca572fd4fdba011ab74dc2b52f277b9 Azure:

[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385226#comment-17385226 ] ASF GitHub Bot commented on HUDI-2176: -- codecov-commenter edited a comment on pull request #3306:

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table

2021-07-21 Thread GitBox
codecov-commenter edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883054849 #

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig

2021-07-21 Thread GitBox
codecov-commenter edited a comment on pull request #3307: URL: https://github.com/apache/hudi/pull/3307#issuecomment-883347621 #

[jira] [Commented] (HUDI-2191) Bump flink version to 1.13.1

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385223#comment-17385223 ] ASF GitHub Bot commented on HUDI-2191: -- garyli1019 commented on pull request #3291: URL:

[GitHub] [hudi] garyli1019 commented on pull request #3291: [HUDI-2191] Bump flink version to 1.13.1

2021-07-21 Thread GitBox
garyli1019 commented on pull request #3291: URL: https://github.com/apache/hudi/pull/3291#issuecomment-884627266 @danny0405 What's the impact of upgrading from 1.12.2 to 1.12.3? we should be very cautious about the version upgrade unless they are fully backward compatible. -- This is

[jira] [Commented] (HUDI-2191) Bump flink version to 1.13.1

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385219#comment-17385219 ] ASF GitHub Bot commented on HUDI-2191: -- danny0405 commented on pull request #3291: URL:

[GitHub] [hudi] danny0405 commented on pull request #3291: [HUDI-2191] Bump flink version to 1.13.1

2021-07-21 Thread GitBox
danny0405 commented on pull request #3291: URL: https://github.com/apache/hudi/pull/3291#issuecomment-884625586 > @danny0405 I think we should at least support 1.12 for the 0.9.0 release, if you think this upgrade is necessary, I'd recommend support both 1.12 and 1.13. Maybe we can have a

[jira] [Commented] (HUDI-2187) Hive integration Improvment

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385214#comment-17385214 ] ASF GitHub Bot commented on HUDI-2187: -- hudi-bot edited a comment on pull request #3289: URL:

[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385215#comment-17385215 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3306: URL:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883052706 ## CI report: * 719bb1046e9d69d88395d65d2f928af35fb9d9bd Azure:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3289: [HUDI-2187] Add a shim layer to support multiple hive version

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 1a9c6e49be153e7cf9646da93c1114ae75969c22 Azure:

[jira] [Commented] (HUDI-2187) Hive integration Improvment

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385212#comment-17385212 ] ASF GitHub Bot commented on HUDI-2187: -- hudi-bot edited a comment on pull request #3289: URL:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3289: [HUDI-2187] Add a shim layer to support multiple hive version

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 3df6f8079655b7128a0abc9b362133396f7c89e8 Azure:

[jira] [Commented] (HUDI-2187) Hive integration Improvment

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385211#comment-17385211 ] ASF GitHub Bot commented on HUDI-2187: -- hudi-bot edited a comment on pull request #3289: URL:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3289: [HUDI-2187] Add a shim layer to support multiple hive version

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 3df6f8079655b7128a0abc9b362133396f7c89e8 Azure:

[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385209#comment-17385209 ] ASF GitHub Bot commented on HUDI-2176: -- codecov-commenter edited a comment on pull request #3306:

[jira] [Commented] (HUDI-2187) Hive integration Improvment

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385207#comment-17385207 ] ASF GitHub Bot commented on HUDI-2187: -- hudi-bot edited a comment on pull request #3289: URL:

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table

2021-07-21 Thread GitBox
codecov-commenter edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883054849 #

[GitHub] [hudi] hudi-bot edited a comment on pull request #3289: [HUDI-2187] Add a shim layer to support multiple hive version

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3289: URL: https://github.com/apache/hudi/pull/3289#issuecomment-881900670 ## CI report: * 3df6f8079655b7128a0abc9b362133396f7c89e8 Azure:

[jira] [Commented] (HUDI-2191) Bump flink version to 1.13.1

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385205#comment-17385205 ] ASF GitHub Bot commented on HUDI-2191: -- garyli1019 commented on pull request #3291: URL:

[GitHub] [hudi] garyli1019 commented on pull request #3291: [HUDI-2191] Bump flink version to 1.13.1

2021-07-21 Thread GitBox
garyli1019 commented on pull request #3291: URL: https://github.com/apache/hudi/pull/3291#issuecomment-884619431 @danny0405 I think we should at least support 1.12 for the 0.9.0 release, if you think this upgrade is necessary, I'd recommend support both 1.12 and 1.13. Maybe we can have a

[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries

2021-07-21 Thread GitBox
nsivabalan commented on issue #3323: URL: https://github.com/apache/hudi/issues/3323#issuecomment-884618704 my bad. looks like you do have a high preCombine value. Just that the datatype is different from what I have tried. may I know what was the datatype of "ts" column in hudi table

[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385202#comment-17385202 ] ASF GitHub Bot commented on HUDI-1771: -- hudi-bot edited a comment on pull request #3285: URL:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 823eea932ca572fd4fdba011ab74dc2b52f277b9 Azure:

[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385201#comment-17385201 ] ASF GitHub Bot commented on HUDI-1771: -- hudi-bot edited a comment on pull request #3285: URL:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3285: [HUDI-1771] Propagate CDC format for hoodie

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3285: URL: https://github.com/apache/hudi/pull/3285#issuecomment-881141261 ## CI report: * 4660e96db4081115eaa7877b8584466347f78fea UNKNOWN * 823eea932ca572fd4fdba011ab74dc2b52f277b9 Azure:

[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries

2021-07-21 Thread GitBox
nsivabalan commented on issue #3323: URL: https://github.com/apache/hudi/issues/3323#issuecomment-884616158 Hudi uses preCombine to merge records from multiple log blocks fyi, but uses combineAndGetUpdateValue() when merging base file w/ the merged records from log. -- This is an

[GitHub] [hudi] nsivabalan commented on issue #3323: [SUPPORT] Trouble with Point in time, Incremental queries

2021-07-21 Thread GitBox
nsivabalan commented on issue #3323: URL: https://github.com/apache/hudi/issues/3323#issuecomment-884615732 I guess its bcoz, you don't have a higher preCombine value for new commits made. I see that all your preCombine is set to same value. Can you try it out and let me know. my

[GitHub] [hudi] leoluan2009 closed pull request #3307: [MINOR] Refactor hive realtime config to extend from HoodieConfig

2021-07-21 Thread GitBox
leoluan2009 closed pull request #3307: URL: https://github.com/apache/hudi/pull/3307 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Commented] (HUDI-2205) Rollback inflight compaction for flink writer

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385196#comment-17385196 ] ASF GitHub Bot commented on HUDI-2205: -- danny0405 closed pull request #3320: URL:

[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385199#comment-17385199 ] ASF GitHub Bot commented on HUDI-2176: -- codecov-commenter edited a comment on pull request #3306:

[jira] [Commented] (HUDI-2205) Rollback inflight compaction for flink writer

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385197#comment-17385197 ] ASF GitHub Bot commented on HUDI-2205: -- danny0405 opened a new pull request #3320: URL:

[GitHub] [hudi] codecov-commenter edited a comment on pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table

2021-07-21 Thread GitBox
codecov-commenter edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883054849 #

[GitHub] [hudi] danny0405 closed pull request #3320: [HUDI-2205] Rollback inflight compaction for flink writer

2021-07-21 Thread GitBox
danny0405 closed pull request #3320: URL: https://github.com/apache/hudi/pull/3320 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Commented] (HUDI-1771) Propagate CDC format for hoodie

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385193#comment-17385193 ] ASF GitHub Bot commented on HUDI-1771: -- danny0405 commented on a change in pull request #3285: URL:

[GitHub] [hudi] danny0405 commented on a change in pull request #3285: [HUDI-1771] Propagate CDC format for hoodie

2021-07-21 Thread GitBox
danny0405 commented on a change in pull request #3285: URL: https://github.com/apache/hudi/pull/3285#discussion_r674454632 ## File path: hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadInputFormat.java ## @@ -615,24 +621,25 @@ public boolean reachedEnd()

[jira] [Commented] (HUDI-2195) Sync Hive Failed When Execute CTAS In Spark2 And Spark3

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385191#comment-17385191 ] ASF GitHub Bot commented on HUDI-2195: -- pengzhiwei2018 commented on a change in pull request #3299:

[jira] [Commented] (HUDI-2195) Sync Hive Failed When Execute CTAS In Spark2 And Spark3

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385192#comment-17385192 ] ASF GitHub Bot commented on HUDI-2195: -- pengzhiwei2018 commented on a change in pull request #3299:

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3299: [HUDI-2195] Sync Hive Failed When Execute CTAS In Spark2 And Spark3

2021-07-21 Thread GitBox
pengzhiwei2018 commented on a change in pull request #3299: URL: https://github.com/apache/hudi/pull/3299#discussion_r674453822 ## File path: packaging/hudi-spark-bundle/pom.xml ## @@ -367,7 +369,18 @@ curator-recipes ${zk-curator.version} - + + +

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #3299: [HUDI-2195] Sync Hive Failed When Execute CTAS In Spark2 And Spark3

2021-07-21 Thread GitBox
pengzhiwei2018 commented on a change in pull request #3299: URL: https://github.com/apache/hudi/pull/3299#discussion_r674453778 ## File path: packaging/hudi-spark-bundle/pom.xml ## @@ -367,7 +369,18 @@ curator-recipes ${zk-curator.version} - + + +

[GitHub] [hudi] nsivabalan commented on issue #3236: why use avro not parquet in MOR

2021-07-21 Thread GitBox
nsivabalan commented on issue #3236: URL: https://github.com/apache/hudi/issues/3236#issuecomment-884608912 Let us know if you have any more questions. if not, will close the ticket. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] nsivabalan commented on issue #3246: OCC correct problem

2021-07-21 Thread GitBox
nsivabalan commented on issue #3246: URL: https://github.com/apache/hudi/issues/3246#issuecomment-884608726 yes, this is a known limitation I guess. @n3nash : Can you confirm and add more if any. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] nsivabalan commented on issue #3321: [SUPPORT] Setting _hoodie_is_deleted column is not deleting records when using Spark DataSource.

2021-07-21 Thread GitBox
nsivabalan commented on issue #3321: URL: https://github.com/apache/hudi/issues/3321#issuecomment-884608382 I see that you are using bulk_insert operation. Don't think w/ bulk_insert we honor the _hoodie_is_deleted. Can you try setting your operation to "upsert". -- This is an

[GitHub] [hudi] nsivabalan commented on issue #3321: [SUPPORT] Setting _hoodie_is_deleted column is not deleting records when using Spark DataSource.

2021-07-21 Thread GitBox
nsivabalan commented on issue #3321: URL: https://github.com/apache/hudi/issues/3321#issuecomment-884607348 May I know whats the schema of your table. do you set default value for "_hoodie_is_deleted" as false? -- This is an automated message from the Apache Git Service. To respond to

[jira] [Commented] (HUDI-2176) Virutal keys support for COW all operations

2021-07-21 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385182#comment-17385182 ] ASF GitHub Bot commented on HUDI-2176: -- hudi-bot edited a comment on pull request #3306: URL:

[GitHub] [hudi] hudi-bot edited a comment on pull request #3306: [HUDI-2176, 2178, 2179] Adding virtual key support to COW table

2021-07-21 Thread GitBox
hudi-bot edited a comment on pull request #3306: URL: https://github.com/apache/hudi/pull/3306#issuecomment-883052706 ## CI report: * 9fd68774b3721b403bba916fec6b55a3a2bba7ba Azure:

  1   2   3   4   >