[GitHub] [hudi] yanghua merged pull request #2474: [HUDI-1453] Fix NPE using HoodieFlinkStreamer to etl data from kafka to hudi

2021-01-22 Thread GitBox
yanghua merged pull request #2474: URL: https://github.com/apache/hudi/pull/2474 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[hudi] branch master updated: [HUDI-1453] Fix NPE using HoodieFlinkStreamer to etl data from kafka to hudi (#2474)

2021-01-22 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new e302c6b [HUDI-1453] Fix NPE using

[GitHub] [hudi] wangxianghu commented on pull request #2473: [HOTFIX] Revert upgrade flink verison to 1.12.0

2021-01-22 Thread GitBox
wangxianghu commented on pull request #2473: URL: https://github.com/apache/hudi/pull/2473#issuecomment-765805235 > @wangxianghu have you verified that this fix makes the flink path happy? i.e any more fixes to do? I tested it in our dev env, it is ok now

[jira] [Closed] (HUDI-1543) Fix NPE using HoodieFlinkStreamer to etl data from kafka to hudi

2021-01-22 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1543. -- Resolution: Fixed > Fix NPE using HoodieFlinkStreamer to etl data from kafka to hudi >

[GitHub] [hudi] xushiyan opened a new pull request #2477: [MINOR] Use skipTests flag for skip.hudi-spark2.unit.tests property

2021-01-22 Thread GitBox
xushiyan opened a new pull request #2477: URL: https://github.com/apache/hudi/pull/2477 To control the property with master flag `skipTests`, which was used in `scripts/run_travis_tests.sh:25` ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit

[GitHub] [hudi] teeyog commented on a change in pull request #2431: [HUDI-1526]translate the api partitionBy to hoodie.datasource.write.partitionpath.field

2021-01-22 Thread GitBox
teeyog commented on a change in pull request #2431: URL: https://github.com/apache/hudi/pull/2431#discussion_r563026619 ## File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala ## @@ -181,16 +183,33 @@ object

[jira] [Created] (HUDI-1544) Add HoodieFlinkStreamer unit test

2021-01-22 Thread wangxianghu (Jira)
wangxianghu created HUDI-1544: - Summary: Add HoodieFlinkStreamer unit test Key: HUDI-1544 URL: https://issues.apache.org/jira/browse/HUDI-1544 Project: Apache Hudi Issue Type: Test

[GitHub] [hudi] codecov-io edited a comment on pull request #2475: [HUDI-1527] automatically infer the data directory, users only need to specify the table directory

2021-01-22 Thread GitBox
codecov-io edited a comment on pull request #2475: URL: https://github.com/apache/hudi/pull/2475#issuecomment-765495259 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2475?src=pr=h1) Report > Merging [#2475](https://codecov.io/gh/apache/hudi/pull/2475?src=pr=desc) (9c38d02) into

[GitHub] [hudi] codecov-io edited a comment on pull request #2475: [HUDI-1527] automatically infer the data directory, users only need to specify the table directory

2021-01-22 Thread GitBox
codecov-io edited a comment on pull request #2475: URL: https://github.com/apache/hudi/pull/2475#issuecomment-765495259 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2475?src=pr=h1) Report > Merging [#2475](https://codecov.io/gh/apache/hudi/pull/2475?src=pr=desc) (9c38d02) into

[GitHub] [hudi] wangxianghu commented on a change in pull request #2431: [HUDI-1526]translate the api partitionBy to hoodie.datasource.write.partitionpath.field

2021-01-22 Thread GitBox
wangxianghu commented on a change in pull request #2431: URL: https://github.com/apache/hudi/pull/2431#discussion_r563024824 ## File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala ## @@ -181,16 +183,33 @@ object

[GitHub] [hudi] rubenssoto commented on pull request #2475: [HUDI-1527] automatically infer the data directory, users only need to specify the table directory

2021-01-22 Thread GitBox
rubenssoto commented on pull request #2475: URL: https://github.com/apache/hudi/pull/2475#issuecomment-765712586 This is a great and important feature to make Hudi easier for no heavy users. This is an automated message from

[GitHub] [hudi] codecov-io edited a comment on pull request #2477: [MINOR] Use skipTests flag for skip.hudi-spark2.unit.tests property

2021-01-22 Thread GitBox
codecov-io edited a comment on pull request #2477: URL: https://github.com/apache/hudi/pull/2477#issuecomment-765873084 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2477?src=pr=h1) Report > Merging [#2477](https://codecov.io/gh/apache/hudi/pull/2477?src=pr=desc) (dd2198f) into

[GitHub] [hudi] wangxianghu commented on pull request #2431: [HUDI-1526]translate the api partitionBy to hoodie.datasource.write.partitionpath.field

2021-01-22 Thread GitBox
wangxianghu commented on pull request #2431: URL: https://github.com/apache/hudi/pull/2431#issuecomment-765865527 > > Hi @teeyog, thanks for your contribution! > > can you add some tests to verify this change > > @wangxianghu Test has been added Thanks, @teeyog will review

[GitHub] [hudi] codecov-io commented on pull request #2477: [MINOR] Use skipTests flag for skip.hudi-spark2.unit.tests property

2021-01-22 Thread GitBox
codecov-io commented on pull request #2477: URL: https://github.com/apache/hudi/pull/2477#issuecomment-765873084 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2477?src=pr=h1) Report > Merging [#2477](https://codecov.io/gh/apache/hudi/pull/2477?src=pr=desc) (dd2198f) into

[GitHub] [hudi] codecov-io edited a comment on pull request #2477: [MINOR] Use skipTests flag for skip.hudi-spark2.unit.tests property

2021-01-22 Thread GitBox
codecov-io edited a comment on pull request #2477: URL: https://github.com/apache/hudi/pull/2477#issuecomment-765873084 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2477?src=pr=h1) Report > Merging [#2477](https://codecov.io/gh/apache/hudi/pull/2477?src=pr=desc) (dd2198f) into

[GitHub] [hudi] wangxianghu commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…

2021-01-22 Thread GitBox
wangxianghu commented on pull request #2452: URL: https://github.com/apache/hudi/pull/2452#issuecomment-765808446 > can we just have a this implemented as a replace of the partition where all files are replaced by an empty list. cleaner would automatically clean the partition that way.

[GitHub] [hudi] vinothchandar commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…

2021-01-22 Thread GitBox
vinothchandar commented on pull request #2452: URL: https://github.com/apache/hudi/pull/2452#issuecomment-765711222 can we just have a this implemented as a replace of the partition where all files are replaced by an empty list. cleaner would automatically clean the partition that way.

[jira] [Updated] (HUDI-1544) Add unit test against HoodieFlinkStreamer

2021-01-22 Thread wangxianghu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1544: -- Summary: Add unit test against HoodieFlinkStreamer (was: Add HoodieFlinkStreamer unit test) > Add

[jira] [Assigned] (HUDI-1544) Add unit test against HoodieFlinkStreamer

2021-01-22 Thread wangxianghu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu reassigned HUDI-1544: - Assignee: wangxianghu > Add unit test against HoodieFlinkStreamer >

[GitHub] [hudi] wangxianghu commented on pull request #2473: [HOTFIX] Revert upgrade flink verison to 1.12.0

2021-01-22 Thread GitBox
wangxianghu commented on pull request #2473: URL: https://github.com/apache/hudi/pull/2473#issuecomment-765808074 > Good catch. Lets file a JIRA to close the testing gap around this? > > Please merge into master when ready. I will port to 0.7.0 branch to do a RC2 filed here:

[GitHub] [hudi] yanghua commented on a change in pull request #2430: [HUDI-1522] Add a new pipeline for Flink writer

2021-01-22 Thread GitBox
yanghua commented on a change in pull request #2430: URL: https://github.com/apache/hudi/pull/2430#discussion_r562494679 ## File path: hudi-flink/src/main/java/org/apache/hudi/operator/StreamWriteFunction.java ## @@ -0,0 +1,344 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] yanghua commented on a change in pull request #2430: [HUDI-1522] Add a new pipeline for Flink writer

2021-01-22 Thread GitBox
yanghua commented on a change in pull request #2430: URL: https://github.com/apache/hudi/pull/2430#discussion_r562465971 ## File path: hudi-flink/src/main/java/org/apache/hudi/streamer/FlinkStreamerConfig.java ## @@ -0,0 +1,124 @@ +/* + * Licensed to the Apache Software

[GitHub] [hudi] wangxianghu edited a comment on pull request #2472: [hotfix] Add back dependency org.apache.flink:flink-connector-kafka-b…

2021-01-22 Thread GitBox
wangxianghu edited a comment on pull request #2472: URL: https://github.com/apache/hudi/pull/2472#issuecomment-765336295 talked with @danny0405 offline we decide to rollback https://github.com/apache/hudi/pull/2384 new change goes to https://github.com/apache/hudi/pull/2473

[GitHub] [hudi] codecov-io commented on pull request #2472: [hotfix] Add back dependency org.apache.flink:flink-connector-kafka-b…

2021-01-22 Thread GitBox
codecov-io commented on pull request #2472: URL: https://github.com/apache/hudi/pull/2472#issuecomment-765347003 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2472?src=pr=h1) Report > Merging [#2472](https://codecov.io/gh/apache/hudi/pull/2472?src=pr=desc) (0a425c7) into

[GitHub] [hudi] codecov-io edited a comment on pull request #2472: [hotfix] Add back dependency org.apache.flink:flink-connector-kafka-b…

2021-01-22 Thread GitBox
codecov-io edited a comment on pull request #2472: URL: https://github.com/apache/hudi/pull/2472#issuecomment-765347003 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2472?src=pr=h1) Report > Merging [#2472](https://codecov.io/gh/apache/hudi/pull/2472?src=pr=desc) (0a425c7) into

[GitHub] [hudi] codecov-io edited a comment on pull request #2472: [hotfix] Add back dependency org.apache.flink:flink-connector-kafka-b…

2021-01-22 Thread GitBox
codecov-io edited a comment on pull request #2472: URL: https://github.com/apache/hudi/pull/2472#issuecomment-765347003 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] danny0405 opened a new pull request #2472: [hotfix] Add back dependency org.apache.flink:flink-connector-kafka-b…

2021-01-22 Thread GitBox
danny0405 opened a new pull request #2472: URL: https://github.com/apache/hudi/pull/2472 …ase_2.11 ## What is the purpose of the pull request Add back the dependency because the `HoodieFlinkStreamer` needs that. ## Brief change log - Modify pom in hudi and

[GitHub] [hudi] wangxianghu edited a comment on pull request #2472: [hotfix] Add back dependency org.apache.flink:flink-connector-kafka-b…

2021-01-22 Thread GitBox
wangxianghu edited a comment on pull request #2472: URL: https://github.com/apache/hudi/pull/2472#issuecomment-765312721 @vinothchandar can we push this pr to release-0.7.0 ? without this change , the job will throw an ClassNotFoundException: ``` java.lang.NoClassDefFoundError:

[GitHub] [hudi] wangxianghu commented on pull request #2472: [hotfix] Add back dependency org.apache.flink:flink-connector-kafka-b…

2021-01-22 Thread GitBox
wangxianghu commented on pull request #2472: URL: https://github.com/apache/hudi/pull/2472#issuecomment-765336295 go to https://github.com/apache/hudi/pull/2473 This is an automated message from the Apache Git Service. To

[GitHub] [hudi] wangxianghu opened a new pull request #2474: [HUDI-1453]Fix NPE using HoodieFlinkStreamer to etl data from kafka t…

2021-01-22 Thread GitBox
wangxianghu opened a new pull request #2474: URL: https://github.com/apache/hudi/pull/2474 …o hudi ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is

[GitHub] [hudi] wangxianghu commented on pull request #2474: [HUDI-1453]Fix NPE using HoodieFlinkStreamer to etl data from kafka t…

2021-01-22 Thread GitBox
wangxianghu commented on pull request #2474: URL: https://github.com/apache/hudi/pull/2474#issuecomment-765343746 @yanghua please take a look when free This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] yanghua commented on pull request #2472: [hotfix] Add back dependency org.apache.flink:flink-connector-kafka-b…

2021-01-22 Thread GitBox
yanghua commented on pull request #2472: URL: https://github.com/apache/hudi/pull/2472#issuecomment-765294659 @wangxianghu Please help to review. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] wangxianghu commented on a change in pull request #2472: [hotfix] Add back dependency org.apache.flink:flink-connector-kafka-b…

2021-01-22 Thread GitBox
wangxianghu commented on a change in pull request #2472: URL: https://github.com/apache/hudi/pull/2472#discussion_r562540064 ## File path: pom.xml ## @@ -106,6 +106,7 @@ 0.8.0 4.4.1 ${spark2.version} +1.11.2 1.12.0 Review comment: can we simply

[jira] [Created] (HUDI-1543) Fix HoodieFlinkStreamer NPE after HUDI-1511 merged

2021-01-22 Thread wangxianghu (Jira)
wangxianghu created HUDI-1543: - Summary: Fix HoodieFlinkStreamer NPE after HUDI-1511 merged Key: HUDI-1543 URL: https://issues.apache.org/jira/browse/HUDI-1543 Project: Apache Hudi Issue Type:

[jira] [Updated] (HUDI-1543) Fix NPE using HoodieFlinkStreamer to etl data from kafka to hudi

2021-01-22 Thread wangxianghu (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxianghu updated HUDI-1543: -- Summary: Fix NPE using HoodieFlinkStreamer to etl data from kafka to hudi (was: Fix

[GitHub] [hudi] codecov-io edited a comment on pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-01-22 Thread GitBox
codecov-io edited a comment on pull request #2325: URL: https://github.com/apache/hudi/pull/2325#issuecomment-742860619 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2325?src=pr=h1) Report > Merging [#2325](https://codecov.io/gh/apache/hudi/pull/2325?src=pr=desc) (79b994a) into

[GitHub] [hudi] peng-xin commented on issue #2448: [SUPPORT] deltacommit for client 172.16.116.102 already exists

2021-01-22 Thread GitBox
peng-xin commented on issue #2448: URL: https://github.com/apache/hudi/issues/2448#issuecomment-765355270 > @peng-xin : Can you enable hoodie.compact.inline -> true and hoodie.auto.commit -> true. The log files are growing because they need to be compacted and if you set the first config,

[GitHub] [hudi] codecov-io edited a comment on pull request #2449: [HUDI-1528] hudi-sync-tools supports synchronization to remote hive

2021-01-22 Thread GitBox
codecov-io edited a comment on pull request #2449: URL: https://github.com/apache/hudi/pull/2449#issuecomment-760950650 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hudi] wangxianghu opened a new pull request #2473: [HOTFIX] Revert upgrade flink verison to 1.12.0

2021-01-22 Thread GitBox
wangxianghu opened a new pull request #2473: URL: https://github.com/apache/hudi/pull/2473 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of

[GitHub] [hudi] codecov-io edited a comment on pull request #2325: [HUDI-699]Fix CompactionCommand and add unit test for CompactionCommand

2021-01-22 Thread GitBox
codecov-io edited a comment on pull request #2325: URL: https://github.com/apache/hudi/pull/2325#issuecomment-742860619 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2325?src=pr=h1) Report > Merging [#2325](https://codecov.io/gh/apache/hudi/pull/2325?src=pr=desc) (79b994a) into

[GitHub] [hudi] wangxianghu commented on a change in pull request #2472: [hotfix] Add back dependency org.apache.flink:flink-connector-kafka-b…

2021-01-22 Thread GitBox
wangxianghu commented on a change in pull request #2472: URL: https://github.com/apache/hudi/pull/2472#discussion_r562534033 ## File path: packaging/hudi-flink-bundle/pom.xml ## @@ -189,6 +190,12 @@ flink-connector-kafka_${scala.binary.version} compile Review

[GitHub] [hudi] wangxianghu commented on pull request #2472: [hotfix] Add back dependency org.apache.flink:flink-connector-kafka-b…

2021-01-22 Thread GitBox
wangxianghu commented on pull request #2472: URL: https://github.com/apache/hudi/pull/2472#issuecomment-765312721 @vinothchandar can we push this pr to release-0.7.0 ? This is an automated message from the Apache Git

[GitHub] [hudi] wangxianghu commented on pull request #2473: [HOTFIX] Revert upgrade flink verison to 1.12.0

2021-01-22 Thread GitBox
wangxianghu commented on pull request #2473: URL: https://github.com/apache/hudi/pull/2473#issuecomment-765344317 @yanghua please take a look when free This is an automated message from the Apache Git Service. To respond to

[jira] [Assigned] (HUDI-1523) Avoid excessive mkdir calls when creating new files

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-1523: - Assignee: sivabalan narayanan > Avoid excessive mkdir calls when creating new

[jira] [Updated] (HUDI-1523) Avoid excessive mkdir calls when creating new files

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1523: -- Labels: user-support-issues (was: ) > Avoid excessive mkdir calls when creating new

[jira] [Commented] (HUDI-1497) Timeout Exception during getFileStatus()

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270131#comment-17270131 ] sivabalan narayanan commented on HUDI-1497: --- [~vbalaji]: can we close this now w/

[jira] [Assigned] (HUDI-1497) Timeout Exception during getFileStatus()

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-1497: - Assignee: Balaji Varadarajan > Timeout Exception during getFileStatus() >

[jira] [Updated] (HUDI-1381) Schedule compaction based on time elapsed

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1381: -- Labels: pull-request-available user-support-issues (was: pull-request-available) >

[jira] [Updated] (HUDI-1523) Avoid excessive mkdir calls when creating new files

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1523: -- Status: New (was: Open) > Avoid excessive mkdir calls when creating new files >

[jira] [Updated] (HUDI-1117) Add tdunning json library to spark and utilities bundle

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1117: -- Labels: user-support-issues (was: ) > Add tdunning json library to spark and utilities

[jira] [Updated] (HUDI-219) Tabify hudi docker demo page

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-219: - Labels: user-support-issues (was: ) > Tabify hudi docker demo page >

[jira] [Commented] (HUDI-219) Tabify hudi docker demo page

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270180#comment-17270180 ] sivabalan narayanan commented on HUDI-219: -- [~bhasudha]: Is this already taken care or still yet

[GitHub] [hudi] codecov-io edited a comment on pull request #2447: [HUDI-1527] automatically infer the data directory, users only need to specify the table directory

2021-01-22 Thread GitBox
codecov-io edited a comment on pull request #2447: URL: https://github.com/apache/hudi/pull/2447#issuecomment-760949326 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2447?src=pr=h1) Report > Merging [#2447](https://codecov.io/gh/apache/hudi/pull/2447?src=pr=desc) (ebf2c70) into

[GitHub] [hudi] leesf commented on pull request #2447: [HUDI-1527] automatically infer the data directory, users only need to specify the table directory

2021-01-22 Thread GitBox
leesf commented on pull request #2447: URL: https://github.com/apache/hudi/pull/2447#issuecomment-765453573 @teeyog it contains unrelated commits from master, you would use `git rebase -i master`. This is an automated

[jira] [Updated] (HUDI-1310) Corruption Block Handling too slow in S3

2021-01-22 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1310: - Fix Version/s: 0.7.0 > Corruption Block Handling too slow in S3 >

[jira] [Updated] (HUDI-1310) Corruption Block Handling too slow in S3

2021-01-22 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1310: - Component/s: Performance > Corruption Block Handling too slow in S3 >

[jira] [Updated] (HUDI-1310) Corruption Block Handling too slow in S3

2021-01-22 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1310: - Fix Version/s: (was: 0.7.0) 0.8.0 > Corruption Block Handling too slow in

[GitHub] [hudi] vburenin commented on pull request #2450: [HUDI-1538] Try to init class trying different signatures instead of checking its name.

2021-01-22 Thread GitBox
vburenin commented on pull request #2450: URL: https://github.com/apache/hudi/pull/2450#issuecomment-765504810 I messed up with a master branch on with a different PR, will close this one and start over. This is an

[jira] [Updated] (HUDI-1501) Explore providing ways to auto-tune input record size based on incoming payload

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1501: -- Labels: user-support-issues (was: ) > Explore providing ways to auto-tune input record

[jira] [Updated] (HUDI-1496) Seek Error when querying MOR tables in GCP

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1496: -- Labels: user-support-issues (was: ) > Seek Error when querying MOR tables in GCP >

[jira] [Updated] (HUDI-1413) Need binary release of Hudi to distribute tools like hudi-cli.sh and hudi-sync

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1413: -- Labels: user-support-issues (was: ) > Need binary release of Hudi to distribute tools

[jira] [Updated] (HUDI-1383) Incorrect partitions getting hive synced

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1383: -- Status: In Progress (was: Open) > Incorrect partitions getting hive synced >

[GitHub] [hudi] codecov-io edited a comment on pull request #2473: [HOTFIX] Revert upgrade flink verison to 1.12.0

2021-01-22 Thread GitBox
codecov-io edited a comment on pull request #2473: URL: https://github.com/apache/hudi/pull/2473#issuecomment-765405863 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[jira] [Commented] (HUDI-1308) Issues found during testing RFC-15

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270156#comment-17270156 ] sivabalan narayanan commented on HUDI-1308: --- [~vinoth] [~vbalaji]: Can we close this ticket or

[jira] [Updated] (HUDI-1501) Explore providing ways to auto-tune input record size based on incoming payload

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1501: -- Labels: (was: user-support-issues) > Explore providing ways to auto-tune input record

[jira] [Updated] (HUDI-1214) Need ability to set deltastreamer checkpoints when doing Spark datasource writes

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1214: -- Labels: user-support-issues (was: ) > Need ability to set deltastreamer checkpoints

[jira] [Commented] (HUDI-1214) Need ability to set deltastreamer checkpoints when doing Spark datasource writes

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270166#comment-17270166 ] sivabalan narayanan commented on HUDI-1214: --- [~vbalaji]: is this a duplicate of

[jira] [Updated] (HUDI-110) Better defaults for Partition extractor for Spark DataSOurce and DeltaStreamer

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-110: - Labels: user-support-issues (was: ) > Better defaults for Partition extractor for Spark

[jira] [Updated] (HUDI-96) Use Command line options instead of positional arguments when launching spark applications from various CLI commands

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-96?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-96: Labels: newbie pull-request-available user-support-issues (was: newbie

[jira] [Updated] (HUDI-258) Hive Query engine not supporting join queries between RT and RO tables

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-258: - Labels: bug-bash-0.6.0 help-requested user-support-issues (was: bug-bash-0.6.0

[GitHub] [hudi] yanghua commented on pull request #2472: [hotfix] Add back dependency org.apache.flink:flink-connector-kafka-b…

2021-01-22 Thread GitBox
yanghua commented on pull request #2472: URL: https://github.com/apache/hudi/pull/2472#issuecomment-765482271 @wangxianghu Can we close this PR now? This is an automated message from the Apache Git Service. To respond to the

[jira] [Resolved] (HUDI-1497) Timeout Exception during getFileStatus()

2021-01-22 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar resolved HUDI-1497. -- Fix Version/s: 0.7.0 Resolution: Fixed This was due to the leaking reader/writers. Fixed

[jira] [Commented] (HUDI-1311) Writes creating/updating large number of files seeing errors when deleting marker files in S3

2021-01-22 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270229#comment-17270229 ] Vinoth Chandar commented on HUDI-1311: -- I noticed something similar. This was related to the leak as

[jira] [Updated] (HUDI-1311) Writes creating/updating large number of files seeing errors when deleting marker files in S3

2021-01-22 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-1311: - Status: Open (was: New) > Writes creating/updating large number of files seeing errors when

[jira] [Resolved] (HUDI-1311) Writes creating/updating large number of files seeing errors when deleting marker files in S3

2021-01-22 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar resolved HUDI-1311. -- Resolution: Fixed > Writes creating/updating large number of files seeing errors when deleting

[jira] [Assigned] (HUDI-1311) Writes creating/updating large number of files seeing errors when deleting marker files in S3

2021-01-22 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-1311: Assignee: Vinoth Chandar > Writes creating/updating large number of files seeing errors

[GitHub] [hudi] vburenin commented on pull request #2476: [HUDI-1538] Try to init class trying different signatures instead of checking its name

2021-01-22 Thread GitBox
vburenin commented on pull request #2476: URL: https://github.com/apache/hudi/pull/2476#issuecomment-765508771 @yanghua This is a new PR that is based on not diverted master branch. This is an automated message from the

[jira] [Updated] (HUDI-1505) Allow pluggable option to write error records to side table, queue

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1505: -- Labels: user-support-issues (was: ) > Allow pluggable option to write error records to

[jira] [Updated] (HUDI-1499) Support configuration to let user override record-size estimate

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1499: -- Labels: newbie user-support-issues (was: newbie) > Support configuration to let user

[jira] [Assigned] (HUDI-1501) Explore providing ways to auto-tune input record size based on incoming payload

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-1501: - Assignee: sivabalan narayanan > Explore providing ways to auto-tune input record

[jira] [Updated] (HUDI-1452) RocksDB FileSystemView throwing NotSerializableError when embedded timeline server is turned off

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1452: -- Labels: user-support-issues (was: ) > RocksDB FileSystemView throwing

[jira] [Updated] (HUDI-1475) Fix documentation of preCombine to clarify when this API is used by Hudi

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1475: -- Labels: user-support-issues (was: ) > Fix documentation of preCombine to clarify when

[GitHub] [hudi] codecov-io commented on pull request #2473: [HOTFIX] Revert upgrade flink verison to 1.12.0

2021-01-22 Thread GitBox
codecov-io commented on pull request #2473: URL: https://github.com/apache/hudi/pull/2473#issuecomment-765405863 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2473?src=pr=h1) Report > Merging [#2473](https://codecov.io/gh/apache/hudi/pull/2473?src=pr=desc) (35d5773) into

[jira] [Updated] (HUDI-1440) Allow option to override schema when doing spark.write

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1440: -- Labels: user-support-issues (was: ) > Allow option to override schema when doing

[jira] [Updated] (HUDI-1383) Incorrect partitions getting hive synced

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1383: -- Status: Patch Available (was: In Progress) > Incorrect partitions getting hive synced

[jira] [Reopened] (HUDI-1383) Incorrect partitions getting hive synced

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reopened HUDI-1383: --- > Incorrect partitions getting hive synced > > >

[jira] [Resolved] (HUDI-1383) Incorrect partitions getting hive synced

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-1383. --- Fix Version/s: (was: 0.8.0) 0.7.0 Resolution: Fixed >

[jira] [Updated] (HUDI-1383) Incorrect partitions getting hive synced

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1383: -- Labels: pull-request-available user-support-issues (was: pull-request-available) >

[jira] [Updated] (HUDI-1383) Incorrect partitions getting hive synced

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1383: -- Status: Closed (was: Patch Available) > Incorrect partitions getting hive synced >

[jira] [Updated] (HUDI-1363) Provide Option to drop columns after they are used to generate partition or record keys

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1363: -- Labels: user-support-issues (was: pull-request-available user-support-issues) >

[jira] [Updated] (HUDI-1363) Provide Option to drop columns after they are used to generate partition or record keys

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1363: -- Labels: pull-request-available user-support-issues (was: pull-request-available) >

[jira] [Commented] (HUDI-1290) Implement Debezium avro source for Delta Streamer

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270159#comment-17270159 ] sivabalan narayanan commented on HUDI-1290: --- This issue could have some pointers 

[jira] [Updated] (HUDI-1280) Add tool to capture earliest or latest offsets in kafka topics

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1280: -- Labels: user-support-issues (was: ) > Add tool to capture earliest or latest offsets

[jira] [Updated] (HUDI-1279) Update Apache Hudi website docs to clarify the property of record_keys

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1279: -- Labels: user-support-issues (was: ) > Update Apache Hudi website docs to clarify the

[jira] [Updated] (HUDI-1363) Provide Option to drop columns after they are used to generate partition or record keys

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1363: -- Labels: (was: user-support-issues) > Provide Option to drop columns after they are

[jira] [Updated] (HUDI-1271) Add utility scripts to perform Restores

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1271: -- Labels: user-support-issues (was: ) > Add utility scripts to perform Restores >

[jira] [Updated] (HUDI-1235) Default vaiue of KeyGenerator configuration is wrongly documented

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1235: -- Labels: user-support-issues (was: ) > Default vaiue of KeyGenerator configuration is

[jira] [Updated] (HUDI-1210) Update doc to clarify that start timestamp is exclusive for incremental queries

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1210: -- Labels: user-support-issues (was: ) > Update doc to clarify that start timestamp is

[jira] [Updated] (HUDI-1201) HoodieDeltaStreamer: Allow user overrides to read from earliest kafka offset when commit files do not have checkpoint

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1201: -- Labels: user-support-issues (was: ) > HoodieDeltaStreamer: Allow user overrides to

[jira] [Updated] (HUDI-1111) Highlight Hudi guarantees in documentation section of website

2021-01-22 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-: -- Labels: user-support-issues (was: ) > Highlight Hudi guarantees in documentation

  1   2   >