[
https://issues.apache.org/jira/browse/HUDI-625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar updated HUDI-625:
--------------------------------
Description:
[https://github.com/apache/incubator-hudi/issues/1328]
So what's going on here is that each entry (single data field) is estimated to
be around 500-750 bytes in memory and things spill a lot...
{code:java}
20/02/20 23:00:39 INFO ExternalSpillableMap: Estimated Payload size => 760 for
3675605,HoodieRecord{key=HoodieKey { recordKey=3675605 partitionPath=default},
currentLocation='HoodieRecordLocation {instantTime=20200220225748,
fileId=499f8d2c-df6a-4275-9166-3de4ac91f3bf-0}',
newLocation='HoodieRecordLocation {instantTime=20200220225921,
fileId=499f8d2c-df6a-4275-9166-3de4ac91f3bf-0}'} {code}
was:
[https://github.com/apache/incubator-hudi/issues/1328]
> Address performance concerns on DiskBasedMap.get() during upsert of small
> workload
> -----------------------------------------------------------------------------------
>
> Key: HUDI-625
> URL: https://issues.apache.org/jira/browse/HUDI-625
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: Performance, Writer Core
> Reporter: Vinoth Chandar
> Assignee: Vinoth Chandar
> Priority: Major
> Fix For: 0.6.0
>
>
> [https://github.com/apache/incubator-hudi/issues/1328]
>
> So what's going on here is that each entry (single data field) is estimated
> to be around 500-750 bytes in memory and things spill a lot...
> {code:java}
> 20/02/20 23:00:39 INFO ExternalSpillableMap: Estimated Payload size => 760
> for 3675605,HoodieRecord{key=HoodieKey { recordKey=3675605
> partitionPath=default}, currentLocation='HoodieRecordLocation
> {instantTime=20200220225748, fileId=499f8d2c-df6a-4275-9166-3de4ac91f3bf-0}',
> newLocation='HoodieRecordLocation {instantTime=20200220225921,
> fileId=499f8d2c-df6a-4275-9166-3de4ac91f3bf-0}'} {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
