[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #2716: [HUDI-1718] when query incr view of mor table which has Multi level partitions, the query failed

2021-04-10 Thread GitBox
xiarixiaoyao commented on a change in pull request #2716: URL: https://github.com/apache/hudi/pull/2716#discussion_r611126662 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java ## @@ -170,7 +170,7 @@ protected

[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #2761: [HUDI-1676] Support SQL with spark3

2021-04-10 Thread GitBox
xiarixiaoyao commented on a change in pull request #2761: URL: https://github.com/apache/hudi/pull/2761#discussion_r611126561 ## File path: hudi-spark-datasource/hudi-spark3-extensions_2.12/src/main/scala/io/hudi/sql/HudiSpark3SessionExtension.scala ## @@ -0,0 +1,64 @@ +/* +

[GitHub] [hudi] garyli1019 commented on a change in pull request #2716: [HUDI-1718] when query incr view of mor table which has Multi level partitions, the query failed

2021-04-10 Thread GitBox
garyli1019 commented on a change in pull request #2716: URL: https://github.com/apache/hudi/pull/2716#discussion_r611126464 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java ## @@ -170,7 +170,7 @@ protected

[GitHub] [hudi] garyli1019 commented on a change in pull request #2761: [HUDI-1676] Support SQL with spark3

2021-04-10 Thread GitBox
garyli1019 commented on a change in pull request #2761: URL: https://github.com/apache/hudi/pull/2761#discussion_r611121740 ## File path: hudi-spark-datasource/hudi-spark3-extensions_2.12/src/main/scala/io/hudi/sql/HudiSpark3SessionExtension.scala ## @@ -0,0 +1,64 @@ +/* + *

[GitHub] [hudi] garyli1019 commented on a change in pull request #2716: [HUDI-1718] when query incr view of mor table which has Multi level partitions, the query failed

2021-04-10 Thread GitBox
garyli1019 commented on a change in pull request #2716: URL: https://github.com/apache/hudi/pull/2716#discussion_r611120591 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/hive/HoodieCombineHiveInputFormat.java ## @@ -170,7 +170,7 @@ protected

[hudi] branch asf-site updated: Travis CI build asf-site

2021-04-10 Thread vinoth
This is an automated email from the ASF dual-hosted git repository. vinoth pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 9d70ca8 Travis CI build asf-site 9d70ca8 is

[GitHub] [hudi] garyli1019 merged pull request #2804: [Hotfix][javadocs] fix typo.

2021-04-10 Thread GitBox
garyli1019 merged pull request #2804: URL: https://github.com/apache/hudi/pull/2804 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[hudi] branch master updated: [MINOR] fix typo. (#2804)

2021-04-10 Thread garyli
This is an automated email from the ASF dual-hosted git repository. garyli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/master by this push: new b554835 [MINOR] fix typo. (#2804) b554835 is

[hudi] branch asf-site updated: [DOC] update query matrix (#2803)

2021-04-10 Thread garyli
This is an automated email from the ASF dual-hosted git repository. garyli pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 2e65656 [DOC] update query matrix (#2803)

[GitHub] [hudi] garyli1019 merged pull request #2803: [DOC] update query matrix

2021-04-10 Thread GitBox
garyli1019 merged pull request #2803: URL: https://github.com/apache/hudi/pull/2803 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [hudi] codecov-io edited a comment on pull request #2714: [HUDI-1707] Reduces log level for too verbose messages from info to debug level.

2021-04-10 Thread GitBox
codecov-io edited a comment on pull request #2714: URL: https://github.com/apache/hudi/pull/2714#issuecomment-813089246 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2714?src=pr=h1) Report > Merging [#2714](https://codecov.io/gh/apache/hudi/pull/2714?src=pr=desc) (334ac44) into

[GitHub] [hudi] codecov-io edited a comment on pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-10 Thread GitBox
codecov-io edited a comment on pull request #2784: URL: https://github.com/apache/hudi/pull/2784#issuecomment-815166346 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2784?src=pr=h1) Report > Merging [#2784](https://codecov.io/gh/apache/hudi/pull/2784?src=pr=desc) (cc724b8) into

[GitHub] [hudi] codecov-io edited a comment on pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-10 Thread GitBox
codecov-io edited a comment on pull request #2784: URL: https://github.com/apache/hudi/pull/2784#issuecomment-815166346 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2784?src=pr=h1) Report > Merging [#2784](https://codecov.io/gh/apache/hudi/pull/2784?src=pr=desc) (09ac4b4) into

[GitHub] [hudi] ssdong commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-10 Thread GitBox
ssdong commented on a change in pull request #2784: URL: https://github.com/apache/hudi/pull/2784#discussion_r611067880 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/MetadataConversionUtils.java ## @@ -72,10 +74,21 @@ public static

[GitHub] [hudi] ssdong commented on pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-10 Thread GitBox
ssdong commented on pull request #2784: URL: https://github.com/apache/hudi/pull/2784#issuecomment-817174595 Hi @satishkotha, I've updated the PR to reflect the most up-to-date conversations regarding handling different scenarios of requested and inflight cases for insert_overwrite and

[GitHub] [hudi] ssdong commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-10 Thread GitBox
ssdong commented on a change in pull request #2784: URL: https://github.com/apache/hudi/pull/2784#discussion_r611070455 ## File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java ## @@ -245,7 +245,7 @@ public final void reset()

[jira] [Commented] (HUDI-1657) build failed on AArch64, Fedora 33

2021-04-10 Thread writewrite1 (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17318542#comment-17318542 ] writewrite1 commented on HUDI-1657: --- if "mvn --version" is still not showing java 1.8, updating

[GitHub] [hudi] ssdong commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-10 Thread GitBox
ssdong commented on a change in pull request #2784: URL: https://github.com/apache/hudi/pull/2784#discussion_r611068273 ## File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/FileCreateUtils.java ## @@ -157,11 +157,19 @@ public static void

[GitHub] [hudi] ssdong commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-10 Thread GitBox
ssdong commented on a change in pull request #2784: URL: https://github.com/apache/hudi/pull/2784#discussion_r611068142 ## File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java ## @@ -239,14 +227,6 @@ public HoodieTestTable

[GitHub] [hudi] ssdong commented on a change in pull request #2784: [HUDI-1740] Fix insert-overwrite API archival

2021-04-10 Thread GitBox
ssdong commented on a change in pull request #2784: URL: https://github.com/apache/hudi/pull/2784#discussion_r611067880 ## File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/MetadataConversionUtils.java ## @@ -72,10 +74,21 @@ public static

[GitHub] [hudi] wosow closed issue #2676: [SUPPORT] When I used 100,000 data to update 100 million data, The program is stuck

2021-04-10 Thread GitBox
wosow closed issue #2676: URL: https://github.com/apache/hudi/issues/2676 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact

[GitHub] [hudi] wosow edited a comment on issue #2676: [SUPPORT] When I used 100,000 data to update 100 million data, The program is stuck

2021-04-10 Thread GitBox
wosow edited a comment on issue #2676: URL: https://github.com/apache/hudi/issues/2676#issuecomment-817145303 > @wosow : once you respond, can you please remove "awaiting-user-response" label for the issue. If possible add "awaiting-community-help" label. Sorry, I have no permission

[GitHub] [hudi] wosow commented on issue #2676: [SUPPORT] When I used 100,000 data to update 100 million data, The program is stuck

2021-04-10 Thread GitBox
wosow commented on issue #2676: URL: https://github.com/apache/hudi/issues/2676#issuecomment-817145303 > @wosow : once you respond, can you please remove "awaiting-user-response" label for the issue. If possible add "awaiting-community-help" label. well -- This is an automated

[GitHub] [hudi] wosow commented on issue #2676: [SUPPORT] When I used 100,000 data to update 100 million data, The program is stuck

2021-04-10 Thread GitBox
wosow commented on issue #2676: URL: https://github.com/apache/hudi/issues/2676#issuecomment-817145233 > Did you try increasing executor instances to see if the job completes in less time ? > Also, it looks like GlobalBloomIndex is enabled. This is needed only when you want record-key

[GitHub] [hudi] codecov-io edited a comment on pull request #2804: [Hotfix][javadocs] fix typo.

2021-04-10 Thread GitBox
codecov-io edited a comment on pull request #2804: URL: https://github.com/apache/hudi/pull/2804#issuecomment-817144816 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2804?src=pr=h1) Report > Merging [#2804](https://codecov.io/gh/apache/hudi/pull/2804?src=pr=desc) (5853c99) into

[GitHub] [hudi] codecov-io commented on pull request #2804: [Hotfix][javadocs] fix typo.

2021-04-10 Thread GitBox
codecov-io commented on pull request #2804: URL: https://github.com/apache/hudi/pull/2804#issuecomment-817144816 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2804?src=pr=h1) Report > Merging [#2804](https://codecov.io/gh/apache/hudi/pull/2804?src=pr=desc) (5853c99) into

[jira] [Updated] (HUDI-1786) Add option for merge max memory

2021-04-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HUDI-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1786: - Labels: pull-request-available (was: ) > Add option for merge max memory >

[GitHub] [hudi] danny0405 opened a new pull request #2805: [HUDI-1786] Add option for merge max memory

2021-04-10 Thread GitBox
danny0405 opened a new pull request #2805: URL: https://github.com/apache/hudi/pull/2805 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the

[jira] [Created] (HUDI-1786) Add option for merge max memory

2021-04-10 Thread Danny Chen (Jira)
Danny Chen created HUDI-1786: Summary: Add option for merge max memory Key: HUDI-1786 URL: https://issues.apache.org/jira/browse/HUDI-1786 Project: Apache Hudi Issue Type: Task

[GitHub] [hudi] garyli1019 commented on a change in pull request #2722: [HUDI-1722]hive beeline/spark-sql query specified field on mor table occur NPE

2021-04-10 Thread GitBox
garyli1019 commented on a change in pull request #2722: URL: https://github.com/apache/hudi/pull/2722#discussion_r611048584 ## File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeInputFormatUtils.java ## @@ -218,6 +218,13 @@ public static void

[GitHub] [hudi] aditiwari01 edited a comment on issue #2797: [SUPPORT] Can not create a Path from an empty string on unpartitioned table

2021-04-10 Thread GitBox
aditiwari01 edited a comment on issue #2797: URL: https://github.com/apache/hudi/issues/2797#issuecomment-817123936 Issue (https://github.com/apache/hudi/issues/2801) might be a duplicate. However while creating an unpartitioned table, my dataframe.write is getting succeeded but I

[GitHub] [hudi] aditiwari01 commented on issue #2801: Issues with Hive querying on MOR tables with no partitions

2021-04-10 Thread GitBox
aditiwari01 commented on issue #2801: URL: https://github.com/apache/hudi/issues/2801#issuecomment-817139358 Please ignore the above comment. The empty result from hive was due to me missinng one of the configs. Everything is working as expected after changing the default value of

[GitHub] [hudi] RocMarshal opened a new pull request #2804: [Hotfix][javadocs] fix typo.

2021-04-10 Thread GitBox
RocMarshal opened a new pull request #2804: URL: https://github.com/apache/hudi/pull/2804 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the

[GitHub] [hudi] garyli1019 opened a new pull request #2803: [DOC] update query matrix

2021-04-10 Thread GitBox
garyli1019 opened a new pull request #2803: URL: https://github.com/apache/hudi/pull/2803 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the

[GitHub] [hudi] nsivabalan closed issue #2498: [SUPPORT] Hudi MERGE_ON_READ load to dataframe fails for the versions [0.6.0],[0.7.0] and runs for [0.5.3]

2021-04-10 Thread GitBox
nsivabalan closed issue #2498: URL: https://github.com/apache/hudi/issues/2498 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [hudi] aditiwari01 opened a new issue #2802: Hive read issues when different partition have different schemas.

2021-04-10 Thread GitBox
aditiwari01 opened a new issue #2802: URL: https://github.com/apache/hudi/issues/2802 Hive reads writer schema separately for each partition. If a schema has evolved and updates has not made for all partitions (i.e. for some partition last change was from older schema), they hive read for

[GitHub] [hudi] aditiwari01 edited a comment on issue #2797: [SUPPORT] Can not create a Path from an empty string on unpartitioned table

2021-04-10 Thread GitBox
aditiwari01 edited a comment on issue #2797: URL: https://github.com/apache/hudi/issues/2797#issuecomment-817123936 Issue (https://github.com/apache/hudi/issues/2801) might be a duplicate. However while creating an unpartitioned table, my dataframe.write is getting succeeded but I

[GitHub] [hudi] aditiwari01 commented on issue #2797: [SUPPORT] Can not create a Path from an empty string on unpartitioned table

2021-04-10 Thread GitBox
aditiwari01 commented on issue #2797: URL: https://github.com/apache/hudi/issues/2797#issuecomment-817123936 Issue (https://github.com/apache/hudi/issues/2801) might be a duplicate. However while creating an unpartitioned table, my dataframe.write is getting succeeded but I am not

[GitHub] [hudi] aditiwari01 commented on issue #2801: Issues with Hive querying on MOR tables with no partitions

2021-04-10 Thread GitBox
aditiwari01 commented on issue #2801: URL: https://github.com/apache/hudi/issues/2801#issuecomment-817120596 Update: After setting default to 0 in case of no partition, I am able to avoid above error and able to get Table meta correctly. However, now select * query on hive return

[GitHub] [hudi] aditiwari01 opened a new issue #2801: Issues with Hive querying on MOR tables with no partitions

2021-04-10 Thread GitBox
aditiwari01 opened a new issue #2801: URL: https://github.com/apache/hudi/issues/2801 Unable to read data via Hive from both _ro & _rt tables if my data is not partitioned. Reading from spark api works fine. Related Write Confs used: ``` PARTITIONPATH_FIELD_OPT_KEY->"",

[GitHub] [hudi] codecov-io edited a comment on pull request #2800: [MINOR] Remove unused imports and some other checkstyle issues

2021-04-10 Thread GitBox
codecov-io edited a comment on pull request #2800: URL: https://github.com/apache/hudi/pull/2800#issuecomment-817093176 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2800?src=pr=h1) Report > Merging [#2800](https://codecov.io/gh/apache/hudi/pull/2800?src=pr=desc) (4ad6646) into

[GitHub] [hudi] codecov-io edited a comment on pull request #2800: [MINOR] Remove unused imports and some other checkstyle issues

2021-04-10 Thread GitBox
codecov-io edited a comment on pull request #2800: URL: https://github.com/apache/hudi/pull/2800#issuecomment-817093176 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2800?src=pr=h1) Report > Merging [#2800](https://codecov.io/gh/apache/hudi/pull/2800?src=pr=desc) (4ad6646) into

[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #2761: [HUDI-1676] Support SQL with spark3

2021-04-10 Thread GitBox
xiarixiaoyao commented on a change in pull request #2761: URL: https://github.com/apache/hudi/pull/2761#discussion_r611020232 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala ## @@ -302,6 +302,10 @@ case class

[GitHub] [hudi] wangxianghu commented on pull request #2800: [MINOR] Remove unused imports and some other checkstyle issues

2021-04-10 Thread GitBox
wangxianghu commented on pull request #2800: URL: https://github.com/apache/hudi/pull/2800#issuecomment-817100662 > it is so weird the unused imports would build success since there is `` in checkstyle.xml yes... seems only `hudi-integ-test` module has this problem -- This is an

[GitHub] [hudi] wangxianghu commented on pull request #2800: [MINOR] Remove unused imports and some other checkstyle issues

2021-04-10 Thread GitBox
wangxianghu commented on pull request #2800: URL: https://github.com/apache/hudi/pull/2800#issuecomment-817100531 reopen to trigger the ci -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hudi] wangxianghu closed pull request #2800: [MINOR] Remove unused imports and some other checkstyle issues

2021-04-10 Thread GitBox
wangxianghu closed pull request #2800: URL: https://github.com/apache/hudi/pull/2800 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [hudi] leesf commented on a change in pull request #2761: [HUDI-1676] Support SQL with spark3

2021-04-10 Thread GitBox
leesf commented on a change in pull request #2761: URL: https://github.com/apache/hudi/pull/2761#discussion_r611015757 ## File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieFileIndex.scala ## @@ -302,6 +302,10 @@ case class HoodieFileIndex(

[GitHub] [hudi] codecov-io commented on pull request #2800: [MINOR] Remove unused imports and some other checkstyle issues

2021-04-10 Thread GitBox
codecov-io commented on pull request #2800: URL: https://github.com/apache/hudi/pull/2800#issuecomment-817093176 # [Codecov](https://codecov.io/gh/apache/hudi/pull/2800?src=pr=h1) Report > Merging [#2800](https://codecov.io/gh/apache/hudi/pull/2800?src=pr=desc) (4ad6646) into

[GitHub] [hudi] wangxianghu opened a new pull request #2800: [MINOR] Remove unused imports and some other checkstyle issues

2021-04-10 Thread GitBox
wangxianghu opened a new pull request #2800: URL: https://github.com/apache/hudi/pull/2800 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of

[GitHub] [hudi] wangxianghu commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition

2021-04-10 Thread GitBox
wangxianghu commented on pull request #2452: URL: https://github.com/apache/hudi/pull/2452#issuecomment-817089012 > can we just have a this implemented as a replace of the partition where all files are replaced by an empty list. cleaner would automatically clean the partition that way.

[GitHub] [hudi] wangxianghu edited a comment on pull request #1946: [HUDI-1176]Upgrade tp log4j2

2021-04-10 Thread GitBox
wangxianghu edited a comment on pull request #1946: URL: https://github.com/apache/hudi/pull/1946#issuecomment-817085349 > @wangxianghu: had upgrade to `2.13.3` and fix the warning. Hi @hddong thanks for your patient. we still need `log4j2.properties` or `log4j2.xml` configs

[GitHub] [hudi] wangxianghu commented on pull request #1946: [HUDI-1176]Upgrade tp log4j2

2021-04-10 Thread GitBox
wangxianghu commented on pull request #1946: URL: https://github.com/apache/hudi/pull/1946#issuecomment-817085349 > @wangxianghu: had upgrade to `2.13.3` and fix the warning. Hi @hddong thanks for your patient. we still need `log4j2.properties` configs right? -- This is an