[jira] [Created] (HUDI-624) Split some of the code from PR for HUDI-479

Suneel Marthi created HUDI-624: -- Summary: Split some of the code from PR for HUDI-479 Key: HUDI-624 URL: https://issues.apache.org/jira/browse/HUDI-624 Project: Apache Hudi (incubating) Issue

[jira] [Updated] (HUDI-624) Split some of the code from PR for HUDI-479

[ https://issues.apache.org/jira/browse/HUDI-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suneel Marthi updated HUDI-624: --- Status: Patch Available (was: In Progress) > Split some of the code from PR for HUDI-479 >

[jira] [Updated] (HUDI-624) Split some of the code from PR for HUDI-479

[ https://issues.apache.org/jira/browse/HUDI-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suneel Marthi updated HUDI-624: --- Status: In Progress (was: Open) > Split some of the code from PR for HUDI-479 >

[jira] [Updated] (HUDI-624) Split some of the code from PR for HUDI-479

[ https://issues.apache.org/jira/browse/HUDI-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suneel Marthi updated HUDI-624: --- Status: Open (was: New) > Split some of the code from PR for HUDI-479 >

[jira] [Created] (HUDI-625) Address performance concerns on DiskBasedMap.get() during upsert of small workload

Vinoth Chandar created HUDI-625: --- Summary: Address performance concerns on DiskBasedMap.get() during upsert of small workload Key: HUDI-625 URL: https://issues.apache.org/jira/browse/HUDI-625 Project:

[jira] [Updated] (HUDI-624) Split some of the code from PR for HUDI-479

[ https://issues.apache.org/jira/browse/HUDI-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-624: Labels: patch pull-request-available (was: patch) > Split some of the code from PR for HUDI-479 >

[GitHub] [incubator-hudi] smarthi opened a new pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479

smarthi opened a new pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479 URL: https://github.com/apache/incubator-hudi/pull/1344 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html

[GitHub] [incubator-hudi] vinothchandar commented on issue #1328: Hudi upsert hangs

vinothchandar commented on issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-589152895 @lamber-ken is right.. I am looking into why the DiskBasedMap is so slow (there was a recent change.. wondering if its a regression.. ) Will raise a

[jira] [Assigned] (HUDI-53) Implement Record level Index to map a record key to a pair #90

[ https://issues.apache.org/jira/browse/HUDI-53?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-53: -- Assignee: sivabalan narayanan (was: Vinoth Chandar) > Implement Record level Index to map a

[jira] [Assigned] (HUDI-145) Limit the amount of partitions considered for GlobalBloomIndex

[ https://issues.apache.org/jira/browse/HUDI-145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-145: --- Assignee: (was: Vinoth Chandar) > Limit the amount of partitions considered for

[jira] [Updated] (HUDI-539) RO Path filter does not pick up hadoop configs from the spark context

[ https://issues.apache.org/jira/browse/HUDI-539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-539: Summary: RO Path filter does not pick up hadoop configs from the spark context (was: No FileSystem

[GitHub] [incubator-hudi] vinothchandar commented on issue #1328: Hudi upsert hangs

vinothchandar commented on issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-589195840 https://issues.apache.org/jira/browse/HUDI-625 filed this to look into this scenario.. @bwu2 In the meantime, could you run your benchmark

[jira] [Created] (HUDI-626) Hudi CLI add export to table option

satish created HUDI-626: --- Summary: Hudi CLI add export to table option Key: HUDI-626 URL: https://issues.apache.org/jira/browse/HUDI-626 Project: Apache Hudi (incubating) Issue Type: Improvement

[GitHub] [incubator-hudi] bvaradar commented on a change in pull request #1150: [HUDI-288]: Add support for ingesting multiple kafka streams in a single DeltaStreamer deployment

bvaradar commented on a change in pull request #1150: [HUDI-288]: Add support for ingesting multiple kafka streams in a single DeltaStreamer deployment URL: https://github.com/apache/incubator-hudi/pull/1150#discussion_r382175651 ## File path:

[jira] [Updated] (HUDI-626) Hudi CLI add export to table option

[ https://issues.apache.org/jira/browse/HUDI-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] satish updated HUDI-626: Description: CLI shell is very restrictive and it is sometimes hard to filter specific rows. Adding ability to

[incubator-hudi] branch master updated: Refactoring getter to avoid double extrametadata in json representation

This is an automated email from the ASF dual-hosted git repository. vbalaji pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git The following commit(s) were added to refs/heads/master by this push: new 185ff64 Refactoring getter to avoid

[jira] [Updated] (HUDI-573) Rolling stats written twice onto commit metadata

[ https://issues.apache.org/jira/browse/HUDI-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-573: Labels: pull-request-available (was: ) > Rolling stats written twice onto commit metadata >

[jira] [Commented] (HUDI-623) Remove UpgradePayloadFromUberToApache

[ https://issues.apache.org/jira/browse/HUDI-623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041219#comment-17041219 ] Vinoth Chandar commented on HUDI-623: - Might be good to leave this around for few more releases? in

[GitHub] [incubator-hudi] satishkotha commented on a change in pull request #1341: [HUDI-626] Add exportToTable option to CLI

satishkotha commented on a change in pull request #1341: [HUDI-626] Add exportToTable option to CLI URL: https://github.com/apache/incubator-hudi/pull/1341#discussion_r382195220 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/utils/TempTableUtil.java ## @@ -0,0

[GitHub] [incubator-hudi] satishkotha commented on a change in pull request #1341: [HUDI-626] Add exportToTable option to CLI

satishkotha commented on a change in pull request #1341: [HUDI-626] Add exportToTable option to CLI URL: https://github.com/apache/incubator-hudi/pull/1341#discussion_r382194382 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/utils/TempTableUtil.java ## @@ -0,0

[jira] [Assigned] (HUDI-295) Do one-time cleanup of Hudi git history

[ https://issues.apache.org/jira/browse/HUDI-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar reassigned HUDI-295: --- Assignee: (was: Vinoth Chandar) > Do one-time cleanup of Hudi git history >

[jira] [Updated] (HUDI-295) Do one-time cleanup of Hudi git history

[ https://issues.apache.org/jira/browse/HUDI-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-295: Status: New (was: Open) > Do one-time cleanup of Hudi git history >

[jira] [Commented] (HUDI-53) Implement Record level Index to map a record key to a pair #90

[ https://issues.apache.org/jira/browse/HUDI-53?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041138#comment-17041138 ] Vinoth Chandar commented on HUDI-53: can we use this for the indexing work? if you have a new one,

[GitHub] [incubator-hudi] bvaradar commented on a change in pull request #1278: [HUDI-573] Refactoring getter to avoid double extrametadata in json representation of HoodieCommitMetadata

bvaradar commented on a change in pull request #1278: [HUDI-573] Refactoring getter to avoid double extrametadata in json representation of HoodieCommitMetadata URL: https://github.com/apache/incubator-hudi/pull/1278#discussion_r382159855 ## File path:

[GitHub] [incubator-hudi] bvaradar merged pull request #1278: [HUDI-573] Refactoring getter to avoid double extrametadata in json representation of HoodieCommitMetadata

bvaradar merged pull request #1278: [HUDI-573] Refactoring getter to avoid double extrametadata in json representation of HoodieCommitMetadata URL: https://github.com/apache/incubator-hudi/pull/1278 This is an automated

[jira] [Closed] (HUDI-573) Rolling stats written twice onto commit metadata

[ https://issues.apache.org/jira/browse/HUDI-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar closed HUDI-573. --- Resolution: Fixed > Rolling stats written twice onto commit metadata >

[jira] [Updated] (HUDI-573) Rolling stats written twice onto commit metadata

[ https://issues.apache.org/jira/browse/HUDI-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-573: Fix Version/s: (was: 0.6.0) 0.5.2 > Rolling stats written twice onto commit

[jira] [Updated] (HUDI-573) Rolling stats written twice onto commit metadata

[ https://issues.apache.org/jira/browse/HUDI-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-573: Status: Open (was: New) > Rolling stats written twice onto commit metadata >

[GitHub] [incubator-hudi] vinothchandar commented on issue #954: org.apache.hudi.org.apache.hadoop_hive.metastore.api.NoSuchObjectException: table not found

vinothchandar commented on issue #954: org.apache.hudi.org.apache.hadoop_hive.metastore.api.NoSuchObjectException: table not found URL: https://github.com/apache/incubator-hudi/issues/954#issuecomment-589242784 @umehrot2 For some of the misconfigs, we could add it to the troubleshooting

[GitHub] [incubator-hudi] satishkotha commented on issue #1341: [HUDI-626] Add exportToTable option to CLI

satishkotha commented on issue #1341: [HUDI-626] Add exportToTable option to CLI URL: https://github.com/apache/incubator-hudi/pull/1341#issuecomment-589252115 > Please first create a JIRA for the PR. @smarthi My bad. Added.

[GitHub] [incubator-hudi] satishkotha commented on a change in pull request #1341: [HUDI-626] Add exportToTable option to CLI

satishkotha commented on a change in pull request #1341: [HUDI-626] Add exportToTable option to CLI URL: https://github.com/apache/incubator-hudi/pull/1341#discussion_r382193841 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/HoodiePrintHelper.java ## @@ -57,11

[GitHub] [incubator-hudi] satishkotha commented on a change in pull request #1341: [HUDI-626] Add exportToTable option to CLI

satishkotha commented on a change in pull request #1341: [HUDI-626] Add exportToTable option to CLI URL: https://github.com/apache/incubator-hudi/pull/1341#discussion_r382193779 ## File path: hudi-cli/src/main/java/org/apache/hudi/cli/HoodiePrintHelper.java ## @@ -18,13

[GitHub] [incubator-hudi] ramachandranms opened a new pull request #1345: [HUDI-618] Adding unit tests for PriorityBasedFileSystemView

ramachandranms opened a new pull request #1345: [HUDI-618] Adding unit tests for PriorityBasedFileSystemView URL: https://github.com/apache/incubator-hudi/pull/1345 ## What is the purpose of the pull request - This PR is to address the JIRA ticket -

[jira] [Updated] (HUDI-618) Improve unit test coverage for org.apache.hudi.common.table.view. PriorityBasedFileSystemView

[ https://issues.apache.org/jira/browse/HUDI-618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-618: Labels: pull-request-available (was: ) > Improve unit test coverage for

[jira] [Assigned] (HUDI-627) Publish coverage to codecov.io

[ https://issues.apache.org/jira/browse/HUDI-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramachandran M S reassigned HUDI-627: - Assignee: Ramachandran M S > Publish coverage to codecov.io >

[jira] [Created] (HUDI-627) Publish coverage to codecov.io

Ramachandran M S created HUDI-627: - Summary: Publish coverage to codecov.io Key: HUDI-627 URL: https://issues.apache.org/jira/browse/HUDI-627 Project: Apache Hudi (incubating) Issue Type:

[GitHub] [incubator-hudi] vinothchandar commented on issue #1328: Hudi upsert hangs

vinothchandar commented on issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-589452198 @bwu2 Got it.. I think the root issue is that the map is spilling more than needed. I am trying to understand why.. Will update the JIRA as I uncover

[GitHub] [incubator-hudi] bwu2 commented on issue #1328: Hudi upsert hangs

bwu2 commented on issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-589446887 Thanks for your replies! @lamber-ken I will try again with that setting. Does increasing the memory available by setting

[GitHub] [incubator-hudi] smarthi commented on a change in pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479

smarthi commented on a change in pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479 URL: https://github.com/apache/incubator-hudi/pull/1344#discussion_r382368430 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/FSUtils.java ##

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479

yanghua commented on a change in pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479 URL: https://github.com/apache/incubator-hudi/pull/1344#discussion_r382363030 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/FSUtils.java ##

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479

yanghua commented on a change in pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479 URL: https://github.com/apache/incubator-hudi/pull/1344#discussion_r382364978 ## File path: hudi-hive/src/main/java/org/apache/hudi/hive/SchemaDifference.java ##

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479

yanghua commented on a change in pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479 URL: https://github.com/apache/incubator-hudi/pull/1344#discussion_r382364125 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/ObjectSizeCalculator.java

[incubator-hudi] branch master updated: [HUDI-624]: Split some of the code from PR for HUDI-479 (#1344)

This is an automated email from the ASF dual-hosted git repository. vinoyang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git The following commit(s) were added to refs/heads/master by this push: new 078d482 [HUDI-624]: Split some of

[GitHub] [incubator-hudi] yanghua merged pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479

yanghua merged pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479 URL: https://github.com/apache/incubator-hudi/pull/1344 This is an automated message from the Apache Git Service. To respond to the

[jira] [Commented] (HUDI-624) Split some of the code from PR for HUDI-479

[ https://issues.apache.org/jira/browse/HUDI-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041584#comment-17041584 ] vinoyang commented on HUDI-624: --- Done via master branch: 8f6035de4a0486e996647e1246334123aed0c9d6 > Split

[jira] [Updated] (HUDI-624) Split some of the code from PR for HUDI-479

[ https://issues.apache.org/jira/browse/HUDI-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated HUDI-624: -- Status: Closed (was: Patch Available) > Split some of the code from PR for HUDI-479 >

Build failed in Jenkins: hudi-snapshot-deployment-0.5 #195

See Changes: -- [...truncated 2.29 KB...] plexus-classworlds-2.5.2.jar /home/jenkins/tools/maven/apache-maven-3.5.4/conf: logging settings.xml toolchains.xml

[jira] [Commented] (HUDI-623) Remove UpgradePayloadFromUberToApache

[ https://issues.apache.org/jira/browse/HUDI-623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041440#comment-17041440 ] vinoyang commented on HUDI-623: --- OK, let wait for more release cycle. > Remove

[GitHub] [incubator-hudi] smarthi commented on a change in pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479

smarthi commented on a change in pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479 URL: https://github.com/apache/incubator-hudi/pull/1344#discussion_r382368430 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/FSUtils.java ##

[GitHub] [incubator-hudi] yanghua commented on a change in pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479

yanghua commented on a change in pull request #1344: [HUDI-624]: Split some of the code from PR for HUDI-479 URL: https://github.com/apache/incubator-hudi/pull/1344#discussion_r382369350 ## File path: hudi-common/src/main/java/org/apache/hudi/common/util/FSUtils.java ##

[jira] [Updated] (HUDI-625) Address performance concerns on DiskBasedMap.get() during upsert of small workload

[ https://issues.apache.org/jira/browse/HUDI-625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-625: Attachment: image-2020-02-20-23-34-24-155.png > Address performance concerns on DiskBasedMap.get()

[jira] [Updated] (HUDI-625) Address performance concerns on DiskBasedMap.get() during upsert of small workload

[ https://issues.apache.org/jira/browse/HUDI-625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-625: Attachment: image-2020-02-20-23-34-27-466.png > Address performance concerns on DiskBasedMap.get()

[jira] [Commented] (HUDI-625) Address performance concerns on DiskBasedMap.get() during upsert of thin records

[ https://issues.apache.org/jira/browse/HUDI-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041647#comment-17041647 ] Vinoth Chandar commented on HUDI-625: - Following is the test code used to profile these object sizes   

[jira] [Updated] (HUDI-625) Address performance concerns on DiskBasedMap.get() during upsert of thin records

[ https://issues.apache.org/jira/browse/HUDI-625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lamber-ken updated HUDI-625: Description: [https://github.com/apache/incubator-hudi/issues/1328]    So what's going on here is that

[jira] [Commented] (HUDI-625) Address performance concerns on DiskBasedMap.get() during upsert of thin records

[ https://issues.apache.org/jira/browse/HUDI-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041644#comment-17041644 ] lamber-ken commented on HUDI-625: - Thinking several solutions can try: :) * Use Spliterator, instead of

[jira] [Updated] (HUDI-625) Address performance concerns on DiskBasedMap.get() during upsert of small workload

[ https://issues.apache.org/jira/browse/HUDI-625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-625: Description: [https://github.com/apache/incubator-hudi/issues/1328]    So what's going on here is

[jira] [Updated] (HUDI-625) Address performance concerns on DiskBasedMap.get() during upsert of thin records

[ https://issues.apache.org/jira/browse/HUDI-625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lamber-ken updated HUDI-625: Attachment: image-2020-02-21-15-35-56-637.png > Address performance concerns on DiskBasedMap.get() during

[jira] [Updated] (HUDI-625) Address performance concerns on DiskBasedMap.get() during upsert of thin records

[ https://issues.apache.org/jira/browse/HUDI-625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lamber-ken updated HUDI-625: Description: [https://github.com/apache/incubator-hudi/issues/1328]    So what's going on here is that

[jira] [Updated] (HUDI-625) Address performance concerns on DiskBasedMap.get() during upsert of thin records

[ https://issues.apache.org/jira/browse/HUDI-625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-625: Summary: Address performance concerns on DiskBasedMap.get() during upsert of thin records (was:

[jira] [Updated] (HUDI-625) Address performance concerns on DiskBasedMap.get() during upsert of small workload

[ https://issues.apache.org/jira/browse/HUDI-625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-625: Description: [https://github.com/apache/incubator-hudi/issues/1328]    So what's going on here is

[GitHub] [incubator-hudi] lamber-ken commented on issue #1328: Hudi upsert hangs

lamber-ken commented on issue #1328: Hudi upsert hangs URL: https://github.com/apache/incubator-hudi/issues/1328#issuecomment-588971355 Hi @bwu2, add option when upsert `option("hoodie.memory.merge.max.size", "200485760")`, let's try again : )