[GitHub] [hudi] codecov-io edited a comment on pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
codecov-io edited a comment on pull request #2651: URL: https://github.com/apache/hudi/pull/2651#issuecomment-794945140 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2651?src=pr=h1) Report > Merging [#2651](https://codecov.io/gh/apache/hudi/pull/2651?src=pr=desc) (ee4ae96) into

[jira] [Updated] (HUDI-1543) Fix NPE using HoodieFlinkStreamer to etl data from kafka to hudi

2021-03-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1543: - Labels: pull-request-available (was: ) > Fix NPE using HoodieFlinkStreamer to etl data from

[GitHub] [hudi] wangxianghu commented on pull request #2474: [HUDI-1543] Fix NPE using HoodieFlinkStreamer to etl data from kafka to hudi

2021-03-23 Thread GitBox
wangxianghu commented on pull request #2474: URL: https://github.com/apache/hudi/pull/2474#issuecomment-805399186 > @wangxianghu looks like the JIRA ticket number of this commit is inaccurate yes, should be 1543 -- This is an automated message from the Apache Git Service. To

[jira] [Resolved] (HUDI-1710) Read optimized query type for Flink batch reader

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1710. --- Resolution: Implemented > Read optimized query type for Flink batch reader >

[jira] [Closed] (HUDI-1710) Read optimized query type for Flink batch reader

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1710. - > Read optimized query type for Flink batch reader > > >

[jira] [Updated] (HUDI-1310) Corruption Block Handling too slow in S3

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1310: -- Fix Version/s: (was: 0.9.0) 0.8.0 > Corruption Block Handling too slow in S3 >

[jira] [Closed] (HUDI-1310) Corruption Block Handling too slow in S3

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1310. - > Corruption Block Handling too slow in S3 > > > Key:

[GitHub] [hudi] Sugamber commented on issue #2637: [SUPPORT] - Partial Update : update few columns of a table

2021-03-23 Thread GitBox
Sugamber commented on issue #2637: URL: https://github.com/apache/hudi/issues/2637#issuecomment-805503994 @n3nash Please confirm if this use case can be achieved . If yes, provide few inputs. -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [hudi] danny0405 opened a new pull request #2711: [hotfix] Log the error message for creating table source first

2021-03-23 Thread GitBox
danny0405 opened a new pull request #2711: URL: https://github.com/apache/hudi/pull/2711 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the

[jira] [Updated] (HUDI-1310) Corruption Block Handling too slow in S3

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1310: -- Affects Version/s: (was: 0.8.0) > Corruption Block Handling too slow in S3 >

[hudi] branch master updated (0e6909d -> 03668db)

2021-03-23 Thread garyli
This is an automated email from the ASF dual-hosted git repository. garyli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git. from 0e6909d [MINOR][DOCUMENT] Update README doc for integ test (#2703) add 03668db [HUDI-1710] Read optimized query

[GitHub] [hudi] garyli1019 merged pull request #2702: [HUDI-1710] Read optimized query type for Flink batch reader

2021-03-23 Thread GitBox
garyli1019 merged pull request #2702: URL: https://github.com/apache/hudi/pull/2702 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [hudi] stayrascal opened a new issue #2712: [SUPPORT] May I ask how to delete data by Flink SQL

2021-03-23 Thread GitBox
stayrascal opened a new issue #2712: URL: https://github.com/apache/hudi/issues/2712 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)? - Join the mailing list to engage in conversations and get faster

[GitHub] [hudi] liujinhui1994 opened a new pull request #2710: [RFC-20] test

2021-03-23 Thread GitBox
liujinhui1994 opened a new pull request #2710: URL: https://github.com/apache/hudi/pull/2710 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of

[jira] [Commented] (HUDI-1139) Add support for JuiceFS

2021-03-23 Thread Changjian Gao (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307519#comment-17307519 ] Changjian Gao commented on HUDI-1139: - Just FYI, JuiceFS has already open sourced: 

[jira] [Updated] (HUDI-1310) Corruption Block Handling too slow in S3

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1310: -- Affects Version/s: (was: 0.9.0) 0.8.0 > Corruption Block Handling too slow in S3 >

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r599292364 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala ## @@ -0,0 +1,317 @@ +/* + * Licensed to the

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r599297507 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/DefaultSource.scala ## @@ -162,18 +184,26 @@ class DefaultSource

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r599296787 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala ## @@ -0,0 +1,317 @@ +/* + * Licensed to the

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r599303993 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java ## @@ -179,6 +179,9 @@ public static

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r599297333 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala ## @@ -0,0 +1,317 @@ +/* + * Licensed to the

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r599319824 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala ## @@ -0,0 +1,317 @@ +/* + * Licensed to the

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r599318844 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/DefaultSource.scala ## @@ -79,39 +82,58 @@ class DefaultSource extends

[GitHub] [hudi] Sugamber edited a comment on issue #2637: [SUPPORT] - Partial Update : update few columns of a table

2021-03-23 Thread GitBox
Sugamber edited a comment on issue #2637: URL: https://github.com/apache/hudi/issues/2637#issuecomment-804636424 I have created one class after implementing HoodieRecordPayload. We have three methods for which we have to write our logic. 1. preCombine 2. combineAndGetUpdateValue

[GitHub] [hudi] danny0405 closed pull request #2702: [HUDI-1710] Read optimized query type for Flink batch reader

2021-03-23 Thread GitBox
danny0405 closed pull request #2702: URL: https://github.com/apache/hudi/pull/2702 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] aditiwari01 commented on issue #2675: [SUPPORT] Unable to query MOR table after schema evolution

2021-03-23 Thread GitBox
aditiwari01 commented on issue #2675: URL: https://github.com/apache/hudi/issues/2675#issuecomment-804668465 > You can add null as default value for your new field if that would work for you. We can not provide defaults from spark. Need to change where hudi registers the avro

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r599296787 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala ## @@ -0,0 +1,317 @@ +/* + * Licensed to the

[GitHub] [hudi] Sugamber commented on issue #2637: [SUPPORT] - Partial Update : update few columns of a table

2021-03-23 Thread GitBox
Sugamber commented on issue #2637: URL: https://github.com/apache/hudi/issues/2637#issuecomment-804748373 Can this use case be achieved using Hudi? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [hudi] cdmikechen edited a comment on issue #2705: [SUPPORT] Can not read data schema using Spark3.0.2 on k8s with hudi-utilities (build in 2.12 and spark3)

2021-03-23 Thread GitBox
cdmikechen edited a comment on issue #2705: URL: https://github.com/apache/hudi/issues/2705#issuecomment-804636641 I've found the problem: There is a new configuration named `hoodie.deltastreamer.schemaprovider.spark_avro_post_processor.enable` and it is `true` by default. If I use my

[GitHub] [hudi] zherenyu831 commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-03-23 Thread GitBox
zherenyu831 commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-804762627 simple workaround is delete first commit file -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] Sugamber edited a comment on issue #2637: [SUPPORT] - Partial Update : update few columns of a table

2021-03-23 Thread GitBox
Sugamber edited a comment on issue #2637: URL: https://github.com/apache/hudi/issues/2637#issuecomment-804636424 I have created one class after implementing HoodieRecordPayload. We have three methods for which we have to write our logic. 1. preCombine 2. combineAndGetUpdateValue

[GitHub] [hudi] pengzhiwei2018 commented on a change in pull request #2651: [HUDI-1591] [RFC-26] Improve Hoodie Table Query Performance And Ease Of Use Fo…

2021-03-23 Thread GitBox
pengzhiwei2018 commented on a change in pull request #2651: URL: https://github.com/apache/hudi/pull/2651#discussion_r599474180 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala ## @@ -0,0 +1,317 @@ +/* + * Licensed to the

[GitHub] [hudi] Sugamber edited a comment on issue #2637: [SUPPORT] - Partial Update : update few columns of a table

2021-03-23 Thread GitBox
Sugamber edited a comment on issue #2637: URL: https://github.com/apache/hudi/issues/2637#issuecomment-804636424 I have created one class after implementing HoodieRecordPayload. We have three methods for which we have to write our logic. 1. preCombine 2. combineAndGetUpdateValue

[GitHub] [hudi] zherenyu831 commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-03-23 Thread GitBox
zherenyu831 commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-804831651 But got new error ``` User class threw exception: org.apache.hudi.exception.HoodieIOException: Could not read commit details from

[GitHub] [hudi] zherenyu831 opened a new issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-03-23 Thread GitBox
zherenyu831 opened a new issue #2707: URL: https://github.com/apache/hudi/issues/2707 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)? - Join the mailing list to engage in conversations and get faster

[GitHub] [hudi] Sugamber edited a comment on issue #2637: [SUPPORT] - Partial Update : update few columns of a table

2021-03-23 Thread GitBox
Sugamber edited a comment on issue #2637: URL: https://github.com/apache/hudi/issues/2637#issuecomment-804748373 Can this use case be achieved using Hudi as target schema and incremental schema are not same? -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] hddong commented on pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-03-23 Thread GitBox
hddong commented on pull request #2325: URL: https://github.com/apache/hudi/pull/2325#issuecomment-804749723 @yanghua Can you have a review for this again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] hddong commented on pull request #1946: [HUDI-1176]Upgrade tp log4j2

2021-03-23 Thread GitBox
hddong commented on pull request #1946: URL: https://github.com/apache/hudi/pull/1946#issuecomment-804760962 @yanghua @vinothchandar: pls have a review when free. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hudi] zherenyu831 commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-03-23 Thread GitBox
zherenyu831 commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-804849028 Also saw some comment and find below link https://issues.apache.org/jira/browse/HUDI-1518 Seems you are going to use cleaner to delete the file group, but currently, if I

[GitHub] [hudi] bvaradar commented on issue #2707: [SUPPORT] insert_ovewrite_table failed on archiving

2021-03-23 Thread GitBox
bvaradar commented on issue #2707: URL: https://github.com/apache/hudi/issues/2707#issuecomment-80482 @satishkotha : Can you take a look at this ? (cc @n3nash ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hudi] vinothchandar commented on pull request #2697: [HUDI-1211] clean up spark session for each test of FunctionalTestHar…

2021-03-23 Thread GitBox
vinothchandar commented on pull request #2697: URL: https://github.com/apache/hudi/pull/2697#issuecomment-804913961 @kwondw this is done intentionally to speed up tests. cc @xushiyan can you please comment on this ? -- This is an automated message from the Apache Git Service.

[jira] [Updated] (HUDI-1650) Improve AvroKafkaSource that it can take into account the schema evolution

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1650: -- Fix Version/s: 0.8.0 > Improve AvroKafkaSource that it can take into account the schema evolution >

[jira] [Resolved] (HUDI-1650) Improve AvroKafkaSource that it can take into account the schema evolution

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1650. --- Resolution: Fixed > Improve AvroKafkaSource that it can take into account the schema evolution >

[jira] [Resolved] (HUDI-1685) Current date should not be initialized in the constructor

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1685. --- Resolution: Fixed > Current date should not be initialized in the constructor >

[jira] [Closed] (HUDI-1685) Current date should not be initialized in the constructor

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1685. - > Current date should not be initialized in the constructor >

[jira] [Commented] (HUDI-1685) Current date should not be initialized in the constructor

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307102#comment-17307102 ] Gary Li commented on HUDI-1685: --- [~akanungo] thanks for your contribution. I can't assign this ticket to you

[jira] [Updated] (HUDI-1553) Add configs for TimelineServer to configure Jetty

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1553: -- Fix Version/s: 0.8.0 > Add configs for TimelineServer to configure Jetty >

[jira] [Resolved] (HUDI-1553) Add configs for TimelineServer to configure Jetty

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1553. --- Resolution: Resolved > Add configs for TimelineServer to configure Jetty >

[jira] [Closed] (HUDI-1553) Add configs for TimelineServer to configure Jetty

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1553. - > Add configs for TimelineServer to configure Jetty > - > >

[jira] [Resolved] (HUDI-1486) Remove pending rollback and move to cleaner

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1486. --- Resolution: Resolved > Remove pending rollback and move to cleaner >

[jira] [Updated] (HUDI-1539) Bug in HoodieCombineRealtimeRecordReader returns wrong results

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1539: -- Fix Version/s: 0.8.0 > Bug in HoodieCombineRealtimeRecordReader returns wrong results >

[jira] [Closed] (HUDI-1486) Remove pending rollback and move to cleaner

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1486. - > Remove pending rollback and move to cleaner > --- > > Key:

[jira] [Closed] (HUDI-1540) Fix commons-codec shading in hudi-spark-bundle

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1540. - > Fix commons-codec shading in hudi-spark-bundle > -- > >

[jira] [Closed] (HUDI-1420) HoodieTableMetaClient.getMarkerFolderPath works incorrectly on windows client with hdfs server for wrong file seperator

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1420. - > HoodieTableMetaClient.getMarkerFolderPath works incorrectly on windows client > with hdfs server for wrong file

[jira] [Closed] (HUDI-1571) Expose record size info for commits w/ hudi-cli

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1571. - > Expose record size info for commits w/ hudi-cli > --- > >

[jira] [Resolved] (HUDI-1571) Expose record size info for commits w/ hudi-cli

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1571. --- Resolution: Resolved > Expose record size info for commits w/ hudi-cli >

[jira] [Assigned] (HUDI-1650) Improve AvroKafkaSource that it can take into account the schema evolution

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li reassigned HUDI-1650: - Assignee: sivabalan narayanan (was: Volodymyr Burenin) > Improve AvroKafkaSource that it can take into

[jira] [Resolved] (HUDI-1653) Add support for composite keys in NonpartitionedKeyGenerator

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1653. --- Resolution: Fixed > Add support for composite keys in NonpartitionedKeyGenerator >

[jira] [Closed] (HUDI-1653) Add support for composite keys in NonpartitionedKeyGenerator

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1653. - > Add support for composite keys in NonpartitionedKeyGenerator >

[jira] [Commented] (HUDI-1711) Avro Schema Exception with Spark 3.0 in 0.7

2021-03-23 Thread Balaji Varadarajan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307095#comment-17307095 ] Balaji Varadarajan commented on HUDI-1711: -- [~shivnarayan]: Can you triage this issue when you

[GitHub] [hudi] bvaradar commented on issue #2705: [SUPPORT] Can not read data schema using Spark3.0.2 on k8s with hudi-utilities (build in 2.12 and spark3)

2021-03-23 Thread GitBox
bvaradar commented on issue #2705: URL: https://github.com/apache/hudi/issues/2705#issuecomment-804927902 @nsivabalan : This looks like a bug and likely needs fixing. Opened : https://issues.apache.org/jira/browse/HUDI-1711 -- This is an automated message from the Apache Git Service. To

[jira] [Closed] (HUDI-1615) GH Issue 2515/ Failure to archive commits on row writer/delete paths

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1615. - > GH Issue 2515/ Failure to archive commits on row writer/delete paths >

[jira] [Resolved] (HUDI-1615) GH Issue 2515/ Failure to archive commits on row writer/delete paths

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1615. --- Resolution: Fixed > GH Issue 2515/ Failure to archive commits on row writer/delete paths >

[jira] [Resolved] (HUDI-1662) Failed to query real-time view use hive/spark-sql when hudi mor table contains dateType

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1662. --- Resolution: Fixed > Failed to query real-time view use hive/spark-sql when hudi mor table > contains

[jira] [Resolved] (HUDI-1660) Exclude pending compaction & clustering from rollback

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1660. --- Resolution: Fixed > Exclude pending compaction & clustering from rollback >

[jira] [Closed] (HUDI-1660) Exclude pending compaction & clustering from rollback

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1660. - > Exclude pending compaction & clustering from rollback > - > >

[jira] [Updated] (HUDI-1660) Exclude pending compaction & clustering from rollback

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1660: -- Fix Version/s: 0.8.0 > Exclude pending compaction & clustering from rollback >

[jira] [Updated] (HUDI-1583) Hudi will skip remaining log files if there is logFile with zero size in logFileList when merge on read.

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1583: -- Fix Version/s: (was: 0.6.0) 0.8.0 > Hudi will skip remaining log files if there is

[jira] [Updated] (HUDI-1477) Support CopyOnWriteTable in java client

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1477: -- Fix Version/s: 0.8.0 > Support CopyOnWriteTable in java client > --- > >

[jira] [Closed] (HUDI-1477) Support CopyOnWriteTable in java client

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1477. - > Support CopyOnWriteTable in java client > --- > > Key:

[GitHub] [hudi] bvaradar commented on issue #2676: [SUPPORT] When I used 100,000 data to update 100 million data, The program is stuck

2021-03-23 Thread GitBox
bvaradar commented on issue #2676: URL: https://github.com/apache/hudi/issues/2676#issuecomment-804945810 Did you try increasing executor instances to see if the job completes in less time ? Also, it looks like GlobalBloomIndex is enabled. This is needed only when you want record-key

[jira] [Resolved] (HUDI-1545) Add test cases for INSERT_OVERWRITE Operation

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1545. --- Resolution: Fixed > Add test cases for INSERT_OVERWRITE Operation >

[jira] [Updated] (HUDI-1545) Add test cases for INSERT_OVERWRITE Operation

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1545: -- Fix Version/s: 0.8.0 > Add test cases for INSERT_OVERWRITE Operation >

[jira] [Closed] (HUDI-1523) Avoid excessive mkdir calls when creating new files

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1523. - > Avoid excessive mkdir calls when creating new files > --- > >

[jira] [Closed] (HUDI-1519) Improve minKey/maxKey compute in HoodieHFileWriter

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1519. - > Improve minKey/maxKey compute in HoodieHFileWriter > -- > >

[GitHub] [hudi] yanghua merged pull request #2703: [MINOR][DOCUMENT] update README doc for integ test

2021-03-23 Thread GitBox
yanghua merged pull request #2703: URL: https://github.com/apache/hudi/pull/2703 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[hudi] branch master updated: [MINOR][DOCUMENT] Update README doc for integ test (#2703)

2021-03-23 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 0e6909d [MINOR][DOCUMENT] Update README doc

[jira] [Updated] (HUDI-1709) Improve lock config naming convention and add hive metastore uri config

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1709: -- Fix Version/s: 0.8.0 > Improve lock config naming convention and add hive metastore uri config >

[jira] [Resolved] (HUDI-1688) hudi write should uncache rdd, when the write operation is finnished

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1688. --- Resolution: Fixed > hudi write should uncache rdd, when the write operation is finnished >

[jira] [Updated] (HUDI-845) Allow parallel writing to Hudi tables using Locking service

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-845: - Issue Type: New Feature (was: Improvement) > Allow parallel writing to Hudi tables using Locking service >

[GitHub] [hudi] kimberlyamandalu commented on issue #2696: Metadata and runtime exceptions in Hudi 0.7.0 on AWS Glue

2021-03-23 Thread GitBox
kimberlyamandalu commented on issue #2696: URL: https://github.com/apache/hudi/issues/2696#issuecomment-804930443 > @kimberlyamandalu do you have the full stack trace? @vinothchandar there was actually no stack trace in the logs. Those messages I posted above would just repeat for

[jira] [Resolved] (HUDI-1634) Handle the case metadata table cannot be synced due to instants being archived

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1634. --- Resolution: Fixed > Handle the case metadata table cannot be synced due to instants being archived >

[jira] [Closed] (HUDI-1634) Handle the case metadata table cannot be synced due to instants being archived

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1634. - > Handle the case metadata table cannot be synced due to instants being archived >

[jira] [Updated] (HUDI-1634) Handle the case metadata table cannot be synced due to instants being archived

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1634: -- Fix Version/s: 0.8.0 > Handle the case metadata table cannot be synced due to instants being archived >

[jira] [Resolved] (HUDI-1644) Do not delete rollback instants in RollbackActionExecutor

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1644. --- Resolution: Fixed > Do not delete rollback instants in RollbackActionExecutor >

[jira] [Resolved] (HUDI-1582) HiveSyncTool - silently fails (RuntimeException is swallowed)

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1582. --- Resolution: Fixed > HiveSyncTool - silently fails (RuntimeException is swallowed) >

[jira] [Closed] (HUDI-1545) Add test cases for INSERT_OVERWRITE Operation

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1545. - [~slamke] thanks for your contribution. I can't assign this ticket to you since you don't have contributor access

[jira] [Created] (HUDI-1711) Avro Schema Exception with Spark 3.0 in 0.7

2021-03-23 Thread Balaji Varadarajan (Jira)
Balaji Varadarajan created HUDI-1711: Summary: Avro Schema Exception with Spark 3.0 in 0.7 Key: HUDI-1711 URL: https://issues.apache.org/jira/browse/HUDI-1711 Project: Apache Hudi Issue

[jira] [Resolved] (HUDI-845) Allow parallel writing to Hudi tables using Locking service

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-845. -- Resolution: Fixed > Allow parallel writing to Hudi tables using Locking service >

[jira] [Closed] (HUDI-845) Allow parallel writing to Hudi tables using Locking service

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-845. > Allow parallel writing to Hudi tables using Locking service >

[jira] [Updated] (HUDI-1685) Current date should not be initialized in the constructor

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1685: -- Fix Version/s: 0.8.0 > Current date should not be initialized in the constructor >

[jira] [Closed] (HUDI-1587) Add latency and freshness metrics for commits

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1587. - > Add latency and freshness metrics for commits > - > >

[jira] [Resolved] (HUDI-1587) Add latency and freshness metrics for commits

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1587. --- Resolution: Implemented > Add latency and freshness metrics for commits >

[jira] [Assigned] (HUDI-1583) Hudi will skip remaining log files if there is logFile with zero size in logFileList when merge on read.

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li reassigned HUDI-1583: - Assignee: Lietong Liu > Hudi will skip remaining log files if there is logFile with zero size in >

[jira] [Updated] (HUDI-1540) Fix commons-codec shading in hudi-spark-bundle

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1540: -- Fix Version/s: (was: 0.7.0) 0.8.0 > Fix commons-codec shading in hudi-spark-bundle >

[jira] [Resolved] (HUDI-1611) Allow directories to be filtered during the bootstrap of the metadata table

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li resolved HUDI-1611. --- Resolution: Resolved > Allow directories to be filtered during the bootstrap of the metadata table >

[jira] [Closed] (HUDI-1583) Hudi will skip remaining log files if there is logFile with zero size in logFileList when merge on read.

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1583. - > Hudi will skip remaining log files if there is logFile with zero size in > logFileList when merge on read. >

[jira] [Updated] (HUDI-1611) Allow directories to be filtered during the bootstrap of the metadata table

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li updated HUDI-1611: -- Fix Version/s: 0.8.0 > Allow directories to be filtered during the bootstrap of the metadata table >

[jira] [Closed] (HUDI-1611) Allow directories to be filtered during the bootstrap of the metadata table

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Li closed HUDI-1611. - > Allow directories to be filtered during the bootstrap of the metadata table >

[jira] [Commented] (HUDI-1582) HiveSyncTool - silently fails (RuntimeException is swallowed)

2021-03-23 Thread Gary Li (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307123#comment-17307123 ] Gary Li commented on HUDI-1582: --- [~balajeeUber] thanks for your contribution. I can't assign this ticket to

  1   2   3   4   5   6   >