[GitHub] [hudi] shenbinglife opened a new issue #2857: [SUPPORT] How to compile package hudi ?

2021-04-20 Thread GitBox
shenbinglife opened a new issue #2857: URL: https://github.com/apache/hudi/issues/2857 How to compile package hudi ? mvn package -DskipTests -Dskip.tests=true [INFO] Scanning for projects... [INFO] [INFO] < org.apache.hudi:hudi >

[jira] [Assigned] (HUDI-1818) Validate and check the option 'write.precombine.field' for Flink writer

2021-04-20 Thread Jira
[ https://issues.apache.org/jira/browse/HUDI-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 谢波 reassigned HUDI-1818: Assignee: 谢波 > Validate and check the option 'write.precombine.field' for Flink writer > --

[jira] [Created] (HUDI-1818) Validate and check the option 'write.precombine.field' for Flink writer

2021-04-20 Thread Danny Chen (Jira)
Danny Chen created HUDI-1818: Summary: Validate and check the option 'write.precombine.field' for Flink writer Key: HUDI-1818 URL: https://issues.apache.org/jira/browse/HUDI-1818 Project: Apache Hudi

[jira] [Updated] (HUDI-1415) Read Hoodie Table As Spark DataSource Table

2021-04-20 Thread pengzhiwei (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengzhiwei updated HUDI-1415: - Status: Open (was: New) > Read Hoodie Table As Spark DataSource Table >

[jira] [Created] (HUDI-1817) when query incr view of hudi table by using spark-sql. the result is wrong

2021-04-20 Thread tao meng (Jira)
tao meng created HUDI-1817: -- Summary: when query incr view of hudi table by using spark-sql. the result is wrong Key: HUDI-1817 URL: https://issues.apache.org/jira/browse/HUDI-1817 Project: Apache Hudi

[jira] [Created] (HUDI-1816) when query incr view of hudi table by using spark-sql, the query result is wrong

2021-04-20 Thread tao meng (Jira)
tao meng created HUDI-1816: -- Summary: when query incr view of hudi table by using spark-sql, the query result is wrong Key: HUDI-1816 URL: https://issues.apache.org/jira/browse/HUDI-1816 Project: Apache Hudi

[GitHub] [hudi] nsivabalan commented on issue #2830: [SUPPORT]same _hoodie_record_key has duplicates data

2021-04-20 Thread GitBox
nsivabalan commented on issue #2830: URL: https://github.com/apache/hudi/issues/2830#issuecomment-823752462 oh, I see you are using GLOBAL_BLOOM as your index. Can you tell us which version of hudi are you using and other env details. -- This is an automated message from the Apache Git

[GitHub] [hudi] nsivabalan edited a comment on issue #2852: [SUPPORT] Read Hudi Table from Hive - Hive Sync clarification

2021-04-20 Thread GitBox
nsivabalan edited a comment on issue #2852: URL: https://github.com/apache/hudi/issues/2852#issuecomment-823751580 Guess the documentation you have linked actually talks about the usage. ```This will ensure the input format classes with its dependencies are available for query planning

[GitHub] [hudi] nsivabalan commented on issue #2852: [SUPPORT] Read Hudi Table from Hive - Hive Sync clarification

2021-04-20 Thread GitBox
nsivabalan commented on issue #2852: URL: https://github.com/apache/hudi/issues/2852#issuecomment-823751580 Guess the documentation you have linked actually talks about the usage. ```This will ensure the input format classes with its dependencies are available for query planning & exec

[GitHub] [hudi] nsivabalan commented on issue #2855: [SUPPORT] hudi-utilities documentation

2021-04-20 Thread GitBox
nsivabalan commented on issue #2855: URL: https://github.com/apache/hudi/issues/2855#issuecomment-823749990 yes, HoodieDeltastreamer is heavily used by many users, which is in Hudi-utilities-bundle. https://issues.apache.org/jira/browse/HUDI-1815 @bvaradar : can you briefly go over w

[jira] [Updated] (HUDI-1815) Add readme to each bundle to give a brief intro about each bundle

2021-04-20 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1815: -- Labels: docs sev:normal (was: ) > Add readme to each bundle to give a brief intro about

[jira] [Created] (HUDI-1815) Add readme to each bundle to give a brief intro about each bundle

2021-04-20 Thread sivabalan narayanan (Jira)
sivabalan narayanan created HUDI-1815: - Summary: Add readme to each bundle to give a brief intro about each bundle Key: HUDI-1815 URL: https://issues.apache.org/jira/browse/HUDI-1815 Project: Apac

[GitHub] [hudi] nsivabalan commented on issue #2850: [SUPPORT] S3 files skipped by HoodieDeltaStreamer on s3 bucket in continuous mode

2021-04-20 Thread GitBox
nsivabalan commented on issue #2850: URL: https://github.com/apache/hudi/issues/2850#issuecomment-823748401 CC @xushiyan @bvaradar @n3nash -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[GitHub] [hudi] nsivabalan commented on issue #2850: [SUPPORT] S3 files skipped by HoodieDeltaStreamer on s3 bucket in continuous mode

2021-04-20 Thread GitBox
nsivabalan commented on issue #2850: URL: https://github.com/apache/hudi/issues/2850#issuecomment-823747511 We know one bug ATM w/ deltastreamer where if multiple files are present w/ same mod time, deltastreamer could skip some of them. https://issues.apache.org/jira/browse/HUDI-1723 htt

[jira] [Created] (HUDI-1814) Non partitioned table for Flink writer

2021-04-20 Thread Danny Chen (Jira)
Danny Chen created HUDI-1814: Summary: Non partitioned table for Flink writer Key: HUDI-1814 URL: https://issues.apache.org/jira/browse/HUDI-1814 Project: Apache Hudi Issue Type: New Feature

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2853: [HUDI-1812] Add explicit index state TTL option for Flink writer

2021-04-20 Thread GitBox
codecov-commenter edited a comment on pull request #2853: URL: https://github.com/apache/hudi/pull/2853#issuecomment-823113459 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2853?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] wk888 commented on issue #2834: [SUPPORT] Help~~~org.apache.hudi.exception.TableNotFoundException

2021-04-20 Thread GitBox
wk888 commented on issue #2834: URL: https://github.com/apache/hudi/issues/2834#issuecomment-823728826 @yanghua i can find the hoodie file in hdfs: ![image](https://user-images.githubusercontent.com/16316415/115488269-cf2ebd00-a28c-11eb-85ac-73ed631b6f31.png) but from the

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2853: [HUDI-1812] Add explicit index state TTL option for Flink writer

2021-04-20 Thread GitBox
codecov-commenter edited a comment on pull request #2853: URL: https://github.com/apache/hudi/pull/2853#issuecomment-823113459 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2853?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] garyli1019 commented on a change in pull request #2847: [HUDI-1769]Add download page to the site

2021-04-20 Thread GitBox
garyli1019 commented on a change in pull request #2847: URL: https://github.com/apache/hudi/pull/2847#discussion_r617151804 ## File path: docs/_pages/download.cn.md ## @@ -7,29 +7,29 @@ last_modified_at: 2019-12-30T15:59:57-04:00 --- ## Release 0.8.0 -* Source Release : [Ap

[GitHub] [hudi] MyLanPangzi closed pull request #2853: [HUDI-1812] Add explicit index state TTL option for Flink writer

2021-04-20 Thread GitBox
MyLanPangzi closed pull request #2853: URL: https://github.com/apache/hudi/pull/2853 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, pleas

[GitHub] [hudi] xiarixiaoyao commented on pull request #2722: [HUDI-1722]hive beeline/spark-sql query specified field on mor table occur NPE

2021-04-20 Thread GitBox
xiarixiaoyao commented on pull request #2722: URL: https://github.com/apache/hudi/pull/2722#issuecomment-823714627 @lw309637554 @nsivabalan thanks for your review. i will try testHoodieRealtimeCombineHoodieInputFormat in another pr, since it has nothing to do with this problem. --

[jira] [Resolved] (HUDI-1744) [Rollback] rollback fail on mor table when the partition path hasn't any files

2021-04-20 Thread lrz (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lrz resolved HUDI-1744. --- Resolution: Fixed > [Rollback] rollback fail on mor table when the partition path hasn't any files > -

[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #2716: [HUDI-1718] when query incr view of mor table which has Multi level partitions, the query failed

2021-04-20 Thread GitBox
xiarixiaoyao commented on a change in pull request #2716: URL: https://github.com/apache/hudi/pull/2716#discussion_r617137325 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java ## @@ -170,7 +170,7 @@ protected HoodieCombine

[GitHub] [hudi] yanghua commented on issue #2834: [SUPPORT] Help~~~org.apache.hudi.exception.TableNotFoundException

2021-04-20 Thread GitBox
yanghua commented on issue #2834: URL: https://github.com/apache/hudi/issues/2834#issuecomment-823705049 @wk888 OK, I reviewed the code, at `TableNotFoundException.java:53`, the path you provided triggered `FileNotFoundException | IllegalArgumentException`. Did you make sure the path exist

[GitHub] [hudi] yanghua commented on pull request #2853: [HUDI-1812] Add explicit index state TTL option for Flink writer

2021-04-20 Thread GitBox
yanghua commented on pull request #2853: URL: https://github.com/apache/hudi/pull/2853#issuecomment-823702360 Hi @MyLanPangzi Would you please recheck the Travis? If it was not caused by your change, then please retrigger the CI. thanks. -- This is an automated message from the Apache Gi

[GitHub] [hudi] wk888 commented on issue #2834: [SUPPORT] Help~~~org.apache.hudi.exception.TableNotFoundException

2021-04-20 Thread GitBox
wk888 commented on issue #2834: URL: https://github.com/apache/hudi/issues/2834#issuecomment-823700483 @yanghua it seems have no privilege create database not create table and the table is created successed -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] nsivabalan commented on issue #2849: [SUPPORT] - org.apache.hudi.exception.HoodieIOException: Could not load Hoodie properties from file:/tmp/hudi_trips_cow/.hoodie/hoodie.properties

2021-04-20 Thread GitBox
nsivabalan commented on issue #2849: URL: https://github.com/apache/hudi/issues/2849#issuecomment-823641624 Can you clean up the base path once and retry. rm -rf sometimes, there could be some residues. -- This is an automated message from the Apache Git Service. To respond to t

[GitHub] [hudi] nsivabalan commented on pull request #2283: [HUDI-1415] Read Hoodie Table As Spark DataSource Table

2021-04-20 Thread GitBox
nsivabalan commented on pull request #2283: URL: https://github.com/apache/hudi/pull/2283#issuecomment-823619094 great job on the patch 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[GitHub] [hudi] vinothchandar commented on pull request #2283: [HUDI-1415] Read Hoodie Table As Spark DataSource Table

2021-04-20 Thread GitBox
vinothchandar commented on pull request #2283: URL: https://github.com/apache/hudi/pull/2283#issuecomment-823611327 This is a great contribution. Thanks @pengzhiwei2018 ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[hudi] branch master updated: [HUDI-1415] Read Hoodie Table As Spark DataSource Table (#2283)

2021-04-20 Thread uditme
This is an automated email from the ASF dual-hosted git repository. uditme pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new aacb8be [HUDI-1415] Read Hoodie Table As Spark Da

[GitHub] [hudi] umehrot2 merged pull request #2283: [HUDI-1415] Read Hoodie Table As Spark DataSource Table

2021-04-20 Thread GitBox
umehrot2 merged pull request #2283: URL: https://github.com/apache/hudi/pull/2283 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please c

[jira] [Commented] (HUDI-1343) Add standard schema postprocessor which would rewrite the schema using spark-avro conversion

2021-04-20 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325964#comment-17325964 ] sivabalan narayanan commented on HUDI-1343: --- [~liujinhui] [~vbalaji]: Do you fol

[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes

2021-04-20 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325963#comment-17325963 ] Vinoth Chandar commented on HUDI-648: - I see it linked now.  I queued the PR up for re

[GitHub] [hudi] satishkotha commented on pull request #2809: [HUDI-1789] Support reading older snapshots

2021-04-20 Thread GitBox
satishkotha commented on pull request #2809: URL: https://github.com/apache/hudi/pull/2809#issuecomment-823443803 @jsbali added few comments. Can you also check why CI is failing? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

[GitHub] [hudi] satishkotha commented on a change in pull request #2809: [HUDI-1789] Support reading older snapshots

2021-04-20 Thread GitBox
satishkotha commented on a change in pull request #2809: URL: https://github.com/apache/hudi/pull/2809#discussion_r616870583 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieInputFormatUtils.java ## @@ -438,11 +437,20 @@ public static HoodieMetadat

[jira] [Commented] (HUDI-251) JDBC incremental load to HUDI with DeltaStreamer

2021-04-20 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325961#comment-17325961 ] Vinoth Chandar commented on HUDI-251: - Please also feel free to take over the RFC as we

[jira] [Commented] (HUDI-251) JDBC incremental load to HUDI with DeltaStreamer

2021-04-20 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325958#comment-17325958 ] Vinoth Chandar commented on HUDI-251: - On 2, I think we have to enforce some sorting wh

[jira] [Assigned] (HUDI-251) JDBC incremental load to HUDI with DeltaStreamer

2021-04-20 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-251: --- Assignee: Sagar Sumit (was: Purushotham Pushpavanthar) > JDBC incremental load to HUDI with D

[GitHub] [hudi] satishkotha merged pull request #2773: [HUDI-1764] Add Hudi-CLI support for clustering

2021-04-20 Thread GitBox
satishkotha merged pull request #2773: URL: https://github.com/apache/hudi/pull/2773 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, pleas

[hudi] branch master updated: [HUDI-1764] Add Hudi-CLI support for clustering (#2773)

2021-04-20 Thread satish
This is an automated email from the ASF dual-hosted git repository. satish pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new 3253079 [HUDI-1764] Add Hudi-CLI support for clus

[jira] [Commented] (HUDI-1138) Re-implement marker files via timeline server

2021-04-20 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325954#comment-17325954 ] Vinoth Chandar commented on HUDI-1138: -- [~309637554] Please let me know if you are in

[jira] [Comment Edited] (HUDI-1138) Re-implement marker files via timeline server

2021-04-20 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325952#comment-17325952 ] Vinoth Chandar edited comment on HUDI-1138 at 4/20/21, 4:41 PM:

[jira] [Commented] (HUDI-1138) Re-implement marker files via timeline server

2021-04-20 Thread Vinoth Chandar (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325952#comment-17325952 ] Vinoth Chandar commented on HUDI-1138: -- yes. basic idea here is to    0) Maintain t

[jira] [Updated] (HUDI-1652) DiskBasedMap:As time goes by, the number of /temp/***** file handles held by the executor process is increasing

2021-04-20 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1652: -- Status: Patch Available (was: In Progress) > DiskBasedMap:As time goes by, the number o

[jira] [Resolved] (HUDI-1652) DiskBasedMap:As time goes by, the number of /temp/***** file handles held by the executor process is increasing

2021-04-20 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan resolved HUDI-1652. --- Fix Version/s: 0.7.0 Resolution: Fixed > DiskBasedMap:As time goes by, the numb

[jira] [Updated] (HUDI-1652) DiskBasedMap:As time goes by, the number of /temp/***** file handles held by the executor process is increasing

2021-04-20 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1652: -- Status: In Progress (was: Open) > DiskBasedMap:As time goes by, the number of /temp/***

[jira] [Assigned] (HUDI-1652) DiskBasedMap:As time goes by, the number of /temp/***** file handles held by the executor process is increasing

2021-04-20 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reassigned HUDI-1652: - Assignee: Balaji Varadarajan > DiskBasedMap:As time goes by, the number of /temp/

[jira] [Updated] (HUDI-1652) DiskBasedMap:As time goes by, the number of /temp/***** file handles held by the executor process is increasing

2021-04-20 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1652: -- Status: Closed (was: Patch Available) > DiskBasedMap:As time goes by, the number of /te

[jira] [Reopened] (HUDI-1652) DiskBasedMap:As time goes by, the number of /temp/***** file handles held by the executor process is increasing

2021-04-20 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan reopened HUDI-1652: --- > DiskBasedMap:As time goes by, the number of /temp/* file handles held by > the exec

[hudi] branch asf-site updated: Travis CI build asf-site

2021-04-20 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 4f86e1d Travis CI build asf-site 4f86e1d is d

[GitHub] [hudi] nsivabalan commented on a change in pull request #2847: [HUDI-1769]Add download page to the site

2021-04-20 Thread GitBox
nsivabalan commented on a change in pull request #2847: URL: https://github.com/apache/hudi/pull/2847#discussion_r616847950 ## File path: docs/_pages/download.cn.md ## @@ -7,29 +7,29 @@ last_modified_at: 2019-12-30T15:59:57-04:00 --- ## Release 0.8.0 -* Source Release : [Ap

[GitHub] [hudi] nsivabalan commented on a change in pull request #2847: [HUDI-1769]Add download page to the site

2021-04-20 Thread GitBox
nsivabalan commented on a change in pull request #2847: URL: https://github.com/apache/hudi/pull/2847#discussion_r616847043 ## File path: docs/_pages/download.cn.md ## @@ -7,29 +7,29 @@ last_modified_at: 2019-12-30T15:59:57-04:00 --- ## Release 0.8.0 -* Source Release : [Ap

[hudi] branch asf-site updated: [MINOR] Fixing key generators blog content (#2739)

2021-04-20 Thread sivabalan
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 860abd0 [MINOR] Fixing key generators blog

[GitHub] [hudi] nsivabalan merged pull request #2739: [MINOR] Fixing key generators blog content

2021-04-20 Thread GitBox
nsivabalan merged pull request #2739: URL: https://github.com/apache/hudi/pull/2739 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] PavelPetukhov opened a new issue #2856: [SUPPORT] Metrics Prometheus pushgateway

2021-04-20 Thread GitBox
PavelPetukhov opened a new issue #2856: URL: https://github.com/apache/hudi/issues/2856 I have discovered that you've added prometheus related changes like here https://issues.apache.org/jira/browse/HUDI-210 But unfortunately there is no documentation related to pushing hudi metri

[GitHub] [hudi] raphaelauv opened a new issue #2855: [SUPPORT] hudi-utilities documentation

2021-04-20 Thread GitBox
raphaelauv opened a new issue #2855: URL: https://github.com/apache/hudi/issues/2855 **Describe the problem you faced** The hudi-utilities are use in the [Docker Demo](https://hudi.apache.org/docs/docker_demo.html) , but there is no documentation on there purpose and if they can be

[GitHub] [hudi] garyli1019 commented on pull request #2847: [HUDI-1769]Add download page to the site

2021-04-20 Thread GitBox
garyli1019 commented on pull request #2847: URL: https://github.com/apache/hudi/pull/2847#issuecomment-823322655 > @garyli1019 the download links are pointing to dist.apache.org tar balls?? > > while we are at it, can we also update release cwiki page with updating this for each rele

[jira] [Closed] (HUDI-1809) Flink merge on read input split uses wrong base file path for default merge type

2021-04-20 Thread vinoyang (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed HUDI-1809. -- Resolution: Fixed d6d52c60636ae6a0c16469fa6761d0080fddf72f > Flink merge on read input split uses wrong base fi

[hudi] branch master updated: [HUDI-1809] Flink merge on read input split uses wrong base file path for default merge type (#2846)

2021-04-20 Thread vinoyang
This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new d6d52c6 [HUDI-1809] Flink merge on read input s

[GitHub] [hudi] yanghua merged pull request #2846: [HUDI-1809] Flink merge on read input split uses wrong base file path…

2021-04-20 Thread GitBox
yanghua merged pull request #2846: URL: https://github.com/apache/hudi/pull/2846 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please co

[GitHub] [hudi] nsivabalan commented on a change in pull request #2739: [MINOR] Fixing key generators blog content

2021-04-20 Thread GitBox
nsivabalan commented on a change in pull request #2739: URL: https://github.com/apache/hudi/pull/2739#discussion_r616674663 ## File path: docs/_posts/2021-02-13-hudi-key-generators.md ## @@ -5,18 +5,16 @@ author: shivnarayan category: blog --- -Every record in Hudi is uniqu

[GitHub] [hudi] nsivabalan commented on pull request #2767: [HUDI-1761] Adding support for Test your own Schema with QuickStart

2021-04-20 Thread GitBox
nsivabalan commented on pull request #2767: URL: https://github.com/apache/hudi/pull/2767#issuecomment-823266697 if not in source bundle, somewhere in util packages or somewhere would help. For new customers who are looking to try out hudi, would be easy to do sanity check if their schema

[GitHub] [hudi] nsivabalan commented on pull request #2776: [HUDI-1768] spark datasource support schema validate add column

2021-04-20 Thread GitBox
nsivabalan commented on pull request #2776: URL: https://github.com/apache/hudi/pull/2776#issuecomment-823264108 yes, this is still valid. @lw309637554 : ping me here once the PR is ready to be reviewed again. -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [hudi] nsivabalan commented on pull request #2720: [HUDI-1719]hive on spark/mr,Incremental query of the mor table, the partition field is incorrect

2021-04-20 Thread GitBox
nsivabalan commented on pull request #2720: URL: https://github.com/apache/hudi/pull/2720#issuecomment-823254679 @xiarixiaoyao : LGTM. ignore the disabled test for now. can you add a UT for the fix. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] nsivabalan commented on a change in pull request #2716: [HUDI-1718] when query incr view of mor table which has Multi level partitions, the query failed

2021-04-20 Thread GitBox
nsivabalan commented on a change in pull request #2716: URL: https://github.com/apache/hudi/pull/2716#discussion_r616621197 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java ## @@ -170,7 +170,7 @@ protected HoodieCombineFi

[jira] [Commented] (HUDI-1747) Deltastreamer incremental read is not working on the MOR table

2021-04-20 Thread sivabalan narayanan (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325773#comment-17325773 ] sivabalan narayanan commented on HUDI-1747: --- awesome, thanks.  > Deltastreamer

[GitHub] [hudi] nsivabalan commented on a change in pull request #2716: [HUDI-1718] when query incr view of mor table which has Multi level partitions, the query failed

2021-04-20 Thread GitBox
nsivabalan commented on a change in pull request #2716: URL: https://github.com/apache/hudi/pull/2716#discussion_r616621197 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java ## @@ -170,7 +170,7 @@ protected HoodieCombineFi

[GitHub] [hudi] nsivabalan commented on a change in pull request #2716: [HUDI-1718] when query incr view of mor table which has Multi level partitions, the query failed

2021-04-20 Thread GitBox
nsivabalan commented on a change in pull request #2716: URL: https://github.com/apache/hudi/pull/2716#discussion_r616621197 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java ## @@ -170,7 +170,7 @@ protected HoodieCombineFi

[GitHub] [hudi] tooptoop4 commented on issue #2284: [SUPPORT] : Is there a option to achieve SCD 2 in Hudi?

2021-04-20 Thread GitBox
tooptoop4 commented on issue #2284: URL: https://github.com/apache/hudi/issues/2284#issuecomment-823247153 https://aws.amazon.com/blogs/big-data/build-slowly-changing-dimensions-type-2-scd2-with-apache-spark-and-apache-hudi-on-amazon-emr/ -- This is an automated message from the Apache G

[GitHub] [hudi] sbernauer commented on pull request #2012: [HUDI-1129] Deltastreamer Add support for schema evolution

2021-04-20 Thread GitBox
sbernauer commented on pull request #2012: URL: https://github.com/apache/hudi/pull/2012#issuecomment-823214232 @sathyaprakashg @n3nash and others thanks for your work! I have rebased the commit for the current master and resolved all the conflicts here https://github.com/sbernauer/hudi/co

[GitHub] [hudi] nsivabalan commented on pull request #2720: [HUDI-1719]hive on spark/mr,Incremental query of the mor table, the partition field is incorrect

2021-04-20 Thread GitBox
nsivabalan commented on pull request #2720: URL: https://github.com/apache/hudi/pull/2720#issuecomment-823203510 @xiarixiaoyao : I was asking Raymond (@xushiyan ) as to why this test is disabled. From git, I found that he was the one who disabled the test and wanted to get info from him.

[GitHub] [hudi] nsivabalan commented on a change in pull request #2845: [HUDI-1723] Fix path selector listing files with the same mod date

2021-04-20 Thread GitBox
nsivabalan commented on a change in pull request #2845: URL: https://github.com/apache/hudi/pull/2845#discussion_r616593119 ## File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/DFSPathSelector.java ## @@ -121,28 +121,30 @@ public static DFSPathS

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2846: [HUDI-1809] Flink merge on read input split uses wrong base file path…

2021-04-20 Thread GitBox
codecov-commenter edited a comment on pull request #2846: URL: https://github.com/apache/hudi/pull/2846#issuecomment-822274991 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2846?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] danny0405 closed pull request #2846: [HUDI-1809] Flink merge on read input split uses wrong base file path…

2021-04-20 Thread GitBox
danny0405 closed pull request #2846: URL: https://github.com/apache/hudi/pull/2846 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2853: [HUDI-1812] Add explicit index state TTL option for Flink writer

2021-04-20 Thread GitBox
codecov-commenter edited a comment on pull request #2853: URL: https://github.com/apache/hudi/pull/2853#issuecomment-823113459 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2853?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] codecov-commenter commented on pull request #2854: [HUDI-1771] Propagate CDC format for hoodie

2021-04-20 Thread GitBox
codecov-commenter commented on pull request #2854: URL: https://github.com/apache/hudi/pull/2854#issuecomment-823120634 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2854?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache

[GitHub] [hudi] danny0405 commented on a change in pull request #2853: [HUDI-1812] Add explicit index state TTL option for Flink writer

2021-04-20 Thread GitBox
danny0405 commented on a change in pull request #2853: URL: https://github.com/apache/hudi/pull/2853#discussion_r616499589 ## File path: hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java ## @@ -80,6 +80,12 @@ private FlinkOptions() { .defaultValue

[GitHub] [hudi] codecov-commenter edited a comment on pull request #2846: [HUDI-1809] Flink merge on read input split uses wrong base file path…

2021-04-20 Thread GitBox
codecov-commenter edited a comment on pull request #2846: URL: https://github.com/apache/hudi/pull/2846#issuecomment-822274991 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2846?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The

[GitHub] [hudi] nevgin commented on issue #2832: [SUPPORT] Hive on Spark dont work

2021-04-20 Thread GitBox
nevgin commented on issue #2832: URL: https://github.com/apache/hudi/issues/2832#issuecomment-823117798 Directly from the spark, queries are being handled wonderfully. From spark for hive, according to the documentation, I removed the hive*.jar libraries. If you do not delete the hive do

[GitHub] [hudi] codecov-commenter commented on pull request #2853: [HUDI-1812] Add explicit index state TTL option for Flink writer

2021-04-20 Thread GitBox
codecov-commenter commented on pull request #2853: URL: https://github.com/apache/hudi/pull/2853#issuecomment-823113459 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2853?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache

[jira] [Updated] (HUDI-1812) Add explicit index state TTL option for Flink writer

2021-04-20 Thread Jira
[ https://issues.apache.org/jira/browse/HUDI-1812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 谢波 updated HUDI-1812: - Description: Add option: {code:java} public static final ConfigOption INDEX_STATE_TTL = ConfigOptions .key("index.stat

[GitHub] [hudi] danny0405 commented on a change in pull request #2846: [HUDI-1809] Flink merge on read input split uses wrong base file path…

2021-04-20 Thread GitBox
danny0405 commented on a change in pull request #2846: URL: https://github.com/apache/hudi/pull/2846#discussion_r616460746 ## File path: hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadInputFormat.java ## @@ -431,6 +433,10 @@ public void close() { //

[jira] [Updated] (HUDI-1771) Propagate CDC format for hoodie

2021-04-20 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1771: - Labels: pull-request-available (was: ) > Propagate CDC format for hoodie > --

[jira] [Updated] (HUDI-1812) Add explicit index state TTL option for Flink writer

2021-04-20 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1812: - Labels: pull-request-available (was: ) > Add explicit index state TTL option for Flink writer > -

[jira] [Updated] (HUDI-1771) Propagate CDC format for hoodie

2021-04-20 Thread Danny Chen (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-1771: - Summary: Propagate CDC format for hoodie (was: Keep the change flags from CDC source for Flink writer) >

[GitHub] [hudi] danny0405 opened a new pull request #2854: [HUDI-1771] Propagate CDC format for hoodie

2021-04-20 Thread GitBox
danny0405 opened a new pull request #2854: URL: https://github.com/apache/hudi/pull/2854 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the p

[GitHub] [hudi] MyLanPangzi opened a new pull request #2853: [HUDI-1812] Add explicit index state TTL option for Flink writer

2021-04-20 Thread GitBox
MyLanPangzi opened a new pull request #2853: URL: https://github.com/apache/hudi/pull/2853 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the

[jira] [Updated] (HUDI-1812) Add explicit index state TTL option for Flink writer

2021-04-20 Thread Jira
[ https://issues.apache.org/jira/browse/HUDI-1812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 谢波 updated HUDI-1812: - Description: Add option: {code:java} public static final ConfigOption INDEX_STATE_TTL = ConfigOptions .key("index.stat

[GitHub] [hudi] yanghua commented on a change in pull request #2846: [HUDI-1809] Flink merge on read input split uses wrong base file path…

2021-04-20 Thread GitBox
yanghua commented on a change in pull request #2846: URL: https://github.com/apache/hudi/pull/2846#discussion_r616439410 ## File path: hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadInputFormat.java ## @@ -431,6 +433,10 @@ public void close() { // it

[GitHub] [hudi] danny0405 commented on a change in pull request #2846: [HUDI-1809] Flink merge on read input split uses wrong base file path…

2021-04-20 Thread GitBox
danny0405 commented on a change in pull request #2846: URL: https://github.com/apache/hudi/pull/2846#discussion_r616437364 ## File path: hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadInputFormat.java ## @@ -431,6 +433,10 @@ public void close() { //

[GitHub] [hudi] yanghua commented on a change in pull request #2846: [HUDI-1809] Flink merge on read input split uses wrong base file path…

2021-04-20 Thread GitBox
yanghua commented on a change in pull request #2846: URL: https://github.com/apache/hudi/pull/2846#discussion_r616415317 ## File path: hudi-flink/src/main/java/org/apache/hudi/table/format/mor/MergeOnReadInputFormat.java ## @@ -431,6 +433,10 @@ public void close() { // it