[
https://issues.apache.org/jira/browse/HIVE-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656190#action_12656190
]
Joydeep Sen Sarma commented on HIVE-170:
----------------------------------------
one significant optimization we can/should make is when flushing - flush the
entries that have been seen the least often (basically MFU caching). if we have
a count aggregate function we can use that - or if not - might be worth keeping
a count in any case.
default value of hashmaxmemory - perhaps this should be something like
Runtime.getRuntime().maxMemory() - 512m (since we know mappers need some memory
anyway).
i suspect Namit is using number of rows as an escape valve in case the memory
computation is not correct. i do think that we are making life a little
difficult for the user - remember that they not only have to manipulate the
hash map memory - but they also have to size up the jvm spawning options so
that heap size is increased. this is fairly obscure. it also turns out that the
jvm options include non memory related stuff (although in later hadoop options
- memory parameterization is separated out i think) - for example ours looks
like this (-Xmx1024m -Djava.net.preferIPv4Stack=true)
it's worth asking if we can base it on the heap size only. the problem with the
freememory approach was that it didn't account for garbage collection. on the
other hand - we can call gc() every some time that we hit some higher free
memory threshold and then flush out the map everytime we hit some lower free
memory threshold (to guarantee that we dont hit OOM). wouldn't this be easier
for the user (just set jvm option)
this would also be much preferable from the perspective of any future cluster
scheduling based on memory. tasks should just need to specify total memory
required and then figure out how to use it well. schedulers can work off this
memory requirement. if the memory requirement has to be then subdivided into
different components (like hive hashmap) - then things become complicated.
> map-side aggregations does not work properly
> --------------------------------------------
>
> Key: HIVE-170
> URL: https://issues.apache.org/jira/browse/HIVE-170
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: 170.patch, patch2
>
>
> map-side aggregation depends on runtime.freememory() which is not guaranteed
> to return the freeable memory - it depends on when the garbage collector is
> invoked last.
> It might be a good idea to estimate the number of rows that can fit in the
> hash table and then flush the hash table based on that
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
