[jira] [Assigned] (HUDI-118) Provide CLI Option for passing properties to Compactor, Cleaner and ParquetImporter

[ https://issues.apache.org/jira/browse/HUDI-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang reassigned HUDI-118: - Assignee: Pratyaksh Sharma (was: vinoyang) > Provide CLI Option for passing properties to Compactor,

[jira] [Commented] (HUDI-118) Provide CLI Option for passing properties to Compactor, Cleaner and ParquetImporter

[ https://issues.apache.org/jira/browse/HUDI-118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971416#comment-16971416 ] vinoyang commented on HUDI-118: --- Hi [~Pratyaksh], Sorry for the late replay. I am not working on this ticket.

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor)

yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344619136 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/job/HoodieDeltaStreamerWrapper.java ## @@ -0,0 +1,69

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor)

yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344613844 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/writer/DeltaWriter.java ## @@ -0,0 +1,184 @@ +/* + *

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor)

yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344612399 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/writer/AvroDeltaInputWriter.java ## @@ -0,0 +1,119 @@

[GitHub] [incubator-hudi] vinothchandar commented on issue #1001: [HUDI-325] Fix Hive partition error for updated HDFS Hudi table

vinothchandar commented on issue #1001: [HUDI-325] Fix Hive partition error for updated HDFS Hudi table URL: https://github.com/apache/incubator-hudi/pull/1001#issuecomment-552430871 >HDFS partition path provided by Hudi and what it gets from Hive do not match +1 @umehrot2 let

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor)

yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344621141 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/dag/nodes/HiveQueryNode.java ## @@ -0,0 +1,99 @@ +/* +

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor)

yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344612991 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/writer/AvroDeltaInputWriter.java ## @@ -0,0 +1,119 @@

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor)

yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344618562 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/reader/DFSParquetDeltaInputReader.java ## @@ -0,0

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor)

yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344618550 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/reader/DFSParquetDeltaInputReader.java ## @@ -0,0

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor)

yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344614898 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/writer/SparkAvroDeltaInputWriter.java ## @@ -0,0 +1,65

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor)

yanghua commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344619724 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/job/HoodieTestSuiteJob.java ## @@ -0,0 +1,183 @@ +/* +

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #942: [HUDI-137] Fix state transitions for Hudi cleaning action

vinothchandar commented on a change in pull request #942: [HUDI-137] Fix state transitions for Hudi cleaning action URL: https://github.com/apache/incubator-hudi/pull/942#discussion_r344721800 ## File path: hudi-client/src/main/java/org/apache/hudi/HoodieCleanClient.java

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #942: [HUDI-137] Fix state transitions for Hudi cleaning action

vinothchandar commented on a change in pull request #942: [HUDI-137] Fix state transitions for Hudi cleaning action URL: https://github.com/apache/incubator-hudi/pull/942#discussion_r344722272 ## File path: hudi-client/src/main/java/org/apache/hudi/HoodieCleanClient.java

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #942: [HUDI-137] Fix state transitions for Hudi cleaning action

vinothchandar commented on a change in pull request #942: [HUDI-137] Fix state transitions for Hudi cleaning action URL: https://github.com/apache/incubator-hudi/pull/942#discussion_r344722713 ## File path: hudi-client/src/main/java/org/apache/hudi/HoodieWriteClient.java

[GitHub] [incubator-hudi] bhasudha commented on issue #1003: [HUDI-218] Adding Presto support to Integration Test

bhasudha commented on issue #1003: [HUDI-218] Adding Presto support to Integration Test URL: https://github.com/apache/incubator-hudi/pull/1003#issuecomment-552464333 LGTM This is an automated message from the Apache Git

[GitHub] [incubator-hudi] vinothchandar commented on issue #1004: [WIP] [HUDI-15] Adding delete api to HoodieWriteClient

vinothchandar commented on issue #1004: [WIP] [HUDI-15] Adding delete api to HoodieWriteClient URL: https://github.com/apache/incubator-hudi/pull/1004#issuecomment-552464546 @nsivabalan are you planning to break HUDI-15 up into multiple PRs i.e one for write client, datasource and delta

[jira] [Commented] (HUDI-326) Support deleting records with only record_key

[ https://issues.apache.org/jira/browse/HUDI-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971494#comment-16971494 ] Vinoth Chandar commented on HUDI-326: - [~bdscheller] We do have all the commit metadata that can be

[jira] [Assigned] (HUDI-57) Support for writing ORC base files

[ https://issues.apache.org/jira/browse/HUDI-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-57: -- Assignee: Vinoth Chandar (was: pingle wang) > Support for writing ORC base files >

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #994: [HUDI-151] Enable HiveOnSpark queries for RT tables

vinothchandar commented on a change in pull request #994: [HUDI-151] Enable HiveOnSpark queries for RT tables URL: https://github.com/apache/incubator-hudi/pull/994#discussion_r344727985 ## File path:

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #994: [HUDI-151] Enable HiveOnSpark queries for RT tables

vinothchandar commented on a change in pull request #994: [HUDI-151] Enable HiveOnSpark queries for RT tables URL: https://github.com/apache/incubator-hudi/pull/994#discussion_r344729747 ## File path:

[jira] [Closed] (HUDI-218) Add Presto demo commands to hoodie-integ-test/ITTHoodieDemo

[ https://issues.apache.org/jira/browse/HUDI-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar closed HUDI-218. --- Resolution: Fixed > Add Presto demo commands to hoodie-integ-test/ITTHoodieDemo >

[jira] [Updated] (HUDI-253) DeltaStreamer should report nicer error messages for misconfigs

[ https://issues.apache.org/jira/browse/HUDI-253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-253: Status: Closed (was: Patch Available) > DeltaStreamer should report nicer error messages for

[jira] [Commented] (HUDI-326) Support deleting records with only record_key

[ https://issues.apache.org/jira/browse/HUDI-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971521#comment-16971521 ] Vinoth Chandar commented on HUDI-326: - If you only need just record_key, then you'd have to force the

[jira] [Updated] (HUDI-145) Limit the amount of partitions considered for GlobalBloomIndex

[ https://issues.apache.org/jira/browse/HUDI-145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-145: Description: Currently, global bloom index will check inputs against files in all partitions.. In

[GitHub] [incubator-hudi] vinothchandar merged pull request #1003: [HUDI-218] Adding Presto support to Integration Test

vinothchandar merged pull request #1003: [HUDI-218] Adding Presto support to Integration Test URL: https://github.com/apache/incubator-hudi/pull/1003 This is an automated message from the Apache Git Service. To respond to

[incubator-hudi] branch master updated (5f13094 -> 23b303e)

This is an automated email from the ASF dual-hosted git repository. vinoth pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git. from 5f13094 [HUDI-253]: added validations for schema provider class (#995) add 23b303e [HUDI-218] Adding

[incubator-hudi] branch master updated (1483b97 -> 5f13094)

This is an automated email from the ASF dual-hosted git repository. vinoth pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git. from 1483b97 [DOCS] Change Hudi acronyms to plural add 5f13094 [HUDI-253]: added validations for schema

[GitHub] [incubator-hudi] vinothchandar merged pull request #995: [HUDI-253]: added validations for schema provider class

vinothchandar merged pull request #995: [HUDI-253]: added validations for schema provider class URL: https://github.com/apache/incubator-hudi/pull/995 This is an automated message from the Apache Git Service. To respond to

[GitHub] [incubator-hudi] vinothchandar commented on issue #996: Fixes to ensure MOR incr pull provides consistent results

vinothchandar commented on issue #996: Fixes to ensure MOR incr pull provides consistent results URL: https://github.com/apache/incubator-hudi/pull/996#issuecomment-552458472 @n3nash do you have a JIRA for this.. can you please link to PR and title?

[GitHub] [incubator-hudi] vinothchandar commented on issue #991: Hudi Test Suite (Refactor)

vinothchandar commented on issue #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#issuecomment-552463443 @n3nash JIRA please This is an automated message from the Apache Git Service. To

[GitHub] [incubator-hudi] anismiles opened a new issue #1007: Azure Support

anismiles opened a new issue #1007: Azure Support URL: https://github.com/apache/incubator-hudi/issues/1007 Is there a plan to support Azure blobs and Data Lake Storage Gen2? This is an automated message from the Apache Git

[GitHub] [incubator-hudi] bvaradar opened a new pull request #1009: [WIP] [HUDI-308] Avoid Renames for tracking state transitions of all actions on dataset

bvaradar opened a new pull request #1009: [WIP] [HUDI-308] Avoid Renames for tracking state transitions of all actions on dataset URL: https://github.com/apache/incubator-hudi/pull/1009 Contains 2 commits stacked : Please review only (2) for now as there is a separate PR ( #1008 ) for

[jira] [Updated] (HUDI-308) Avoid Renames for tracking state transitions of all actions on dataset

[ https://issues.apache.org/jira/browse/HUDI-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-308: Labels: pull-request-available (was: ) > Avoid Renames for tracking state transitions of all

[jira] [Commented] (HUDI-309) General Redesign of Archived Timeline for efficient scan and management

[ https://issues.apache.org/jira/browse/HUDI-309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971747#comment-16971747 ] Balaji Varadarajan commented on HUDI-309: -

[GitHub] [incubator-hudi] bvaradar commented on a change in pull request #942: [HUDI-137] Fix state transitions for Hudi cleaning action

bvaradar commented on a change in pull request #942: [HUDI-137] Fix state transitions for Hudi cleaning action URL: https://github.com/apache/incubator-hudi/pull/942#discussion_r344825763 ## File path: hudi-client/src/main/java/org/apache/hudi/HoodieCleanClient.java ## @@

[GitHub] [incubator-hudi] bvaradar commented on a change in pull request #942: [HUDI-137] Fix state transitions for Hudi cleaning action

bvaradar commented on a change in pull request #942: [HUDI-137] Fix state transitions for Hudi cleaning action URL: https://github.com/apache/incubator-hudi/pull/942#discussion_r344825531 ## File path: hudi-client/src/main/java/org/apache/hudi/HoodieWriteClient.java ## @@

[jira] [Updated] (HUDI-80) Incrementalize cleaning based on timeline metadata

[ https://issues.apache.org/jira/browse/HUDI-80?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-80: --- Labels: pull-request-available (was: ) > Incrementalize cleaning based on timeline metadata >

[GitHub] [incubator-hudi] bvaradar opened a new pull request #1008: [HUDI-80] Leverage Commit metadata to figure out partitions to be cleaned for Cleaning by commits mode

bvaradar opened a new pull request #1008: [HUDI-80] Leverage Commit metadata to figure out partitions to be cleaned for Cleaning by commits mode URL: https://github.com/apache/incubator-hudi/pull/1008 Additional Details :

[jira] [Resolved] (HUDI-137) Hudi cleaning state changes should be consistent with commit actions

[ https://issues.apache.org/jira/browse/HUDI-137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balaji Varadarajan resolved HUDI-137. - Resolution: Fixed > Hudi cleaning state changes should be consistent with commit actions >

[jira] [Reopened] (HUDI-137) Hudi cleaning state changes should be consistent with commit actions

[ https://issues.apache.org/jira/browse/HUDI-137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balaji Varadarajan reopened HUDI-137: - > Hudi cleaning state changes should be consistent with commit actions >

[jira] [Updated] (HUDI-137) Hudi cleaning state changes should be consistent with commit actions

[ https://issues.apache.org/jira/browse/HUDI-137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balaji Varadarajan updated HUDI-137: Status: Closed (was: Patch Available) > Hudi cleaning state changes should be consistent

[jira] [Closed] (HUDI-137) Hudi cleaning state changes should be consistent with commit actions

[ https://issues.apache.org/jira/browse/HUDI-137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balaji Varadarajan closed HUDI-137. --- > Hudi cleaning state changes should be consistent with commit actions >

[GitHub] [incubator-hudi] yihua commented on a change in pull request #1006: [HUDI-276] Translate the Configurations page into Chinese

yihua commented on a change in pull request #1006: [HUDI-276] Translate the Configurations page into Chinese URL: https://github.com/apache/incubator-hudi/pull/1006#discussion_r344858236 ## File path: docs/configurations.cn.md ## @@ -51,385 +49,419 @@ inputDF.write()

[GitHub] [incubator-hudi] yihua commented on a change in pull request #1006: [HUDI-276] Translate the Configurations page into Chinese

yihua commented on a change in pull request #1006: [HUDI-276] Translate the Configurations page into Chinese URL: https://github.com/apache/incubator-hudi/pull/1006#discussion_r344858247 ## File path: docs/configurations.cn.md ## @@ -51,385 +49,419 @@ inputDF.write()

[incubator-hudi] branch master updated (23b303e -> 1032fc3)

This is an automated email from the ASF dual-hosted git repository. vbalaji pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git. from 23b303e [HUDI-218] Adding Presto support to Integration Test (#1003) add 1032fc3 [HUDI-137] Hudi

[GitHub] [incubator-hudi] bvaradar merged pull request #942: [HUDI-137] Fix state transitions for Hudi cleaning action

bvaradar merged pull request #942: [HUDI-137] Fix state transitions for Hudi cleaning action URL: https://github.com/apache/incubator-hudi/pull/942 This is an automated message from the Apache Git Service. To respond to the

[GitHub] [incubator-hudi] yihua commented on issue #1006: [HUDI-276] Translate the Configurations page into Chinese

yihua commented on issue #1006: [HUDI-276] Translate the Configurations page into Chinese URL: https://github.com/apache/incubator-hudi/pull/1006#issuecomment-552569570 @leesf Thanks for the detailed review. I addressed all your comments in the latest commit.

[jira] [Closed] (HUDI-276) Translate Documentation -> Configurations page

[ https://issues.apache.org/jira/browse/HUDI-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-276. -- Resolution: Fixed This PR is merged: [https://github.com/apache/incubator-hudi/pull/1006] > Translate

[jira] [Updated] (HUDI-277) Translate Documentation -> Performance page

[ https://issues.apache.org/jira/browse/HUDI-277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-277: Labels: pull-request-available (was: ) > Translate Documentation -> Performance page >

[GitHub] [incubator-hudi] yihua opened a new pull request #1010: [HUDI-277] Translate the Performance page into Chinese

yihua opened a new pull request #1010: [HUDI-277] Translate the Performance page into Chinese URL: https://github.com/apache/incubator-hudi/pull/1010 https://issues.apache.org/jira/browse/HUDI-277 This is an automated

[GitHub] [incubator-hudi] vinothchandar commented on issue #1005: [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types

vinothchandar commented on issue #1005: [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types URL: https://github.com/apache/incubator-hudi/pull/1005#issuecomment-552677611 @ezhux

[incubator-hudi] branch asf-site updated: [HUDI-276] Translate the Configurations page into Chinese (#1006)

This is an automated email from the ASF dual-hosted git repository. leesf pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git The following commit(s) were added to refs/heads/asf-site by this push: new 470ff8f [HUDI-276] Translate the

[GitHub] [incubator-hudi] leesf merged pull request #1006: [HUDI-276] Translate the Configurations page into Chinese

leesf merged pull request #1006: [HUDI-276] Translate the Configurations page into Chinese URL: https://github.com/apache/incubator-hudi/pull/1006 This is an automated message from the Apache Git Service. To respond to the

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1005: [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for

vinothchandar commented on a change in pull request #1005: [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types URL: https://github.com/apache/incubator-hudi/pull/1005#discussion_r344968699 ## File

[GitHub] [incubator-hudi] vinothchandar commented on issue #626: Adding documentation for hudi test suite

vinothchandar commented on issue #626: Adding documentation for hudi test suite URL: https://github.com/apache/incubator-hudi/pull/626#issuecomment-552680757 @n3nash we have steadily moved any technical docs to the cwiki. Should this belong there as opposed to the site?

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #961: [HUDI-306] Support Glue catalog and other hive metastore implementations

vinothchandar commented on a change in pull request #961: [HUDI-306] Support Glue catalog and other hive metastore implementations URL: https://github.com/apache/incubator-hudi/pull/961#discussion_r344970164 ## File path: packaging/hudi-spark-bundle/pom.xml ## @@ -144,40

[GitHub] [incubator-hudi] umehrot2 commented on a change in pull request #1005: [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Deci

umehrot2 commented on a change in pull request #1005: [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types URL: https://github.com/apache/incubator-hudi/pull/1005#discussion_r344973529 ## File path:

[GitHub] [incubator-hudi] umehrot2 commented on issue #961: [HUDI-306] Support Glue catalog and other hive metastore implementations

umehrot2 commented on issue #961: [HUDI-306] Support Glue catalog and other hive metastore implementations URL: https://github.com/apache/incubator-hudi/pull/961#issuecomment-552686181 > @umehrot2 More than EMR, I was wondering if we should provide some guidance on how/when to use the

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344979689 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/DeltaInputFormat.java ## @@ -0,0 +1,26 @@ +/* +

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344974625 ## File path: hudi-bench/pom.xml ## @@ -0,0 +1,314 @@ + + +http://www.w3.org/2001/XMLSchema-instance;

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344980862 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/dag/DagUtils.java ## @@ -0,0 +1,247 @@ +/* + *

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344974289 ## File path: docker/demo/config/bench/target.avsc ## @@ -0,0 +1,37 @@ +{ + "type" : "record", + "name" :

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344974875 ## File path: hudi-bench/prepare_integration_suite.sh ## @@ -0,0 +1,112 @@ +#!/bin/bash + +# Determine the

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344974477 ## File path: hudi-bench/pom.xml ## @@ -0,0 +1,314 @@ + + +http://www.w3.org/2001/XMLSchema-instance;

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344979774 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/DeltaOutputType.java ## @@ -0,0 +1,26 @@ +/* + *

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344982472 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/job/HoodieTestSuiteJob.java ## @@ -0,0 +1,183

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344980603 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/configuration/DFSDeltaConfig.java ## @@ -0,0

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344981160 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/dag/WorkflowDag.java ## @@ -0,0 +1,43 @@ +/* + *

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344981709 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/dag/nodes/HiveQueryNode.java ## @@ -0,0 +1,99

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344981580 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/dag/nodes/DagNode.java ## @@ -0,0 +1,125 @@ +/*

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344980291 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/DeltaInputFormat.java ## @@ -0,0 +1,26 @@ +/* +

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344982209 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/generator/GenericRecordFullPayloadGenerator.java

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344978923 ## File path: hudi-bench/prepare_integration_suite.sh ## @@ -0,0 +1,112 @@ +#!/bin/bash + +# Determine the

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344981980 ## File path: hudi-bench/src/main/java/org/apache/hudi/bench/generator/DeltaGenerator.java ## @@ -0,0 +1,236

[jira] [Commented] (HUDI-289) Implement a long running test for Hudi writing and querying end-end

[ https://issues.apache.org/jira/browse/HUDI-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971980#comment-16971980 ] Vinoth Chandar commented on HUDI-289: - [~nishith29][~yanghua] can you please clarify what the plan for

[jira] [Updated] (HUDI-80) Incrementalize cleaning based on timeline metadata

[ https://issues.apache.org/jira/browse/HUDI-80?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-80: --- Status: Patch Available (was: In Progress) > Incrementalize cleaning based on timeline metadata >

[jira] [Updated] (HUDI-12) Upgrade Hudi to Spark 2.4

[ https://issues.apache.org/jira/browse/HUDI-12?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-12: --- Status: Patch Available (was: In Progress) > Upgrade Hudi to Spark 2.4 > - > >

[jira] [Commented] (HUDI-289) Implement a long running test for Hudi writing and querying end-end

[ https://issues.apache.org/jira/browse/HUDI-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971985#comment-16971985 ] vinoyang commented on HUDI-289: --- [~vinoth] IMO, [~nishith29] provided a good infrastructure to do this job.

[GitHub] [incubator-hudi] umehrot2 commented on issue #1005: [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types

umehrot2 commented on issue #1005: [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types URL: https://github.com/apache/incubator-hudi/pull/1005#issuecomment-552682699 > Hi Udit! Thanks for making this PR!

[GitHub] [incubator-hudi] vinothchandar commented on issue #1005: [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types

vinothchandar commented on issue #1005: [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types URL: https://github.com/apache/incubator-hudi/pull/1005#issuecomment-552683915 @umehrot2 I believe we can first

[GitHub] [incubator-hudi] umehrot2 commented on issue #961: [HUDI-306] Support Glue catalog and other hive metastore implementations

umehrot2 commented on issue #961: [HUDI-306] Support Glue catalog and other hive metastore implementations URL: https://github.com/apache/incubator-hudi/pull/961#issuecomment-552684776 > @umehrot2 we moved the build instructions to README. Do you mind adding a EMR build section to the

[GitHub] [incubator-hudi] vinothchandar commented on issue #961: [HUDI-306] Support Glue catalog and other hive metastore implementations

vinothchandar commented on issue #961: [HUDI-306] Support Glue catalog and other hive metastore implementations URL: https://github.com/apache/incubator-hudi/pull/961#issuecomment-552685075 @umehrot2 More than EMR, I was wondering if we should provide some guidance on how/when to use the

[incubator-hudi] branch master updated (1032fc3 -> 0bb5999)

This is an automated email from the ASF dual-hosted git repository. vinoth pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git. from 1032fc3 [HUDI-137] Hudi cleaning state changes should be consistent with compaction actions add

[GitHub] [incubator-hudi] vinothchandar merged pull request #961: [HUDI-306] Support Glue catalog and other hive metastore implementations

vinothchandar merged pull request #961: [HUDI-306] Support Glue catalog and other hive metastore implementations URL: https://github.com/apache/incubator-hudi/pull/961 This is an automated message from the Apache Git

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344979098 ## File path: hudi-bench/prepare_integration_suite.sh ## @@ -0,0 +1,112 @@ +#!/bin/bash + +# Determine the

[jira] [Assigned] (HUDI-12) Upgrade Hudi to Spark 2.4

[ https://issues.apache.org/jira/browse/HUDI-12?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-12: -- Assignee: Udit Mehrotra (was: Vinoth Chandar) > Upgrade Hudi to Spark 2.4 >

[jira] [Updated] (HUDI-106) Dynamically tune bloom filter entries

[ https://issues.apache.org/jira/browse/HUDI-106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-106: Status: Patch Available (was: In Progress) > Dynamically tune bloom filter entries >

[jira] [Updated] (HUDI-25) Faster Incremental queries on Hoodie #492

[ https://issues.apache.org/jira/browse/HUDI-25?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-25: --- Status: Patch Available (was: In Progress) > Faster Incremental queries on Hoodie #492 >

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1008: [HUDI-80] Leverage Commit metadata to figure out partitions to be cleaned for Cleaning by commits mode

vinothchandar commented on a change in pull request #1008: [HUDI-80] Leverage Commit metadata to figure out partitions to be cleaned for Cleaning by commits mode URL: https://github.com/apache/incubator-hudi/pull/1008#discussion_r344984656 ## File path:

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1008: [HUDI-80] Leverage Commit metadata to figure out partitions to be cleaned for Cleaning by commits mode

vinothchandar commented on a change in pull request #1008: [HUDI-80] Leverage Commit metadata to figure out partitions to be cleaned for Cleaning by commits mode URL: https://github.com/apache/incubator-hudi/pull/1008#discussion_r344985313 ## File path:

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1008: [HUDI-80] Leverage Commit metadata to figure out partitions to be cleaned for Cleaning by commits mode

vinothchandar commented on a change in pull request #1008: [HUDI-80] Leverage Commit metadata to figure out partitions to be cleaned for Cleaning by commits mode URL: https://github.com/apache/incubator-hudi/pull/1008#discussion_r344985160 ## File path:

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1008: [HUDI-80] Leverage Commit metadata to figure out partitions to be cleaned for Cleaning by commits mode

vinothchandar commented on a change in pull request #1008: [HUDI-80] Leverage Commit metadata to figure out partitions to be cleaned for Cleaning by commits mode URL: https://github.com/apache/incubator-hudi/pull/1008#discussion_r344984583 ## File path:

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1008: [HUDI-80] Leverage Commit metadata to figure out partitions to be cleaned for Cleaning by commits mode

vinothchandar commented on a change in pull request #1008: [HUDI-80] Leverage Commit metadata to figure out partitions to be cleaned for Cleaning by commits mode URL: https://github.com/apache/incubator-hudi/pull/1008#discussion_r344984837 ## File path:

[jira] [Updated] (HUDI-91) Replace Databricks spark-avro with native spark-avro #628

[ https://issues.apache.org/jira/browse/HUDI-91?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-91: --- Status: Patch Available (was: In Progress) > Replace Databricks spark-avro with native spark-avro #628

[jira] [Closed] (HUDI-312) Investigate recent flaky CI runs

[ https://issues.apache.org/jira/browse/HUDI-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar closed HUDI-312. --- Resolution: Fixed > Investigate recent flaky CI runs > > >

[jira] [Updated] (HUDI-15) Add a delete() API to HoodieWriteClient as well as Spark datasource #531

[ https://issues.apache.org/jira/browse/HUDI-15?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-15: --- Status: Patch Available (was: In Progress) > Add a delete() API to HoodieWriteClient as well as Spark

Build failed in Jenkins: hudi-snapshot-deployment-0.5 #96

See Changes: -- Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on H31 (ubuntu) in workspace

[GitHub] [incubator-hudi] umehrot2 commented on issue #1005: [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types

umehrot2 commented on issue #1005: [HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types URL: https://github.com/apache/incubator-hudi/pull/1005#issuecomment-552683643 > Change looks good. But we need to get

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor)

vinothchandar commented on a change in pull request #991: Hudi Test Suite (Refactor) URL: https://github.com/apache/incubator-hudi/pull/991#discussion_r344973583 ## File path: docker/demo/config/bench/target.avsc ## @@ -0,0 +1,37 @@ +{ + "type" : "record", + "name" :

  1   2   >